Business Segment Reporting - Schedule of Depreciation and Amortization by Segment (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Segment Reporting Information [Line Items]
Depreciation expense	$ 31,911	$ 27,202	$ 26,886
Amortization of mortgage servicing rights			51,788
Amortization of debt discount	1,342	1,183	1,114
Amortization of debt issuance costs	3,170	2,921	2,738
Servicing
Segment Reporting Information [Line Items]
Depreciation expense	1,925	4,601	5,797
Amortization of mortgage servicing rights			51,515
Amortization of debt discount	0	0	0
Amortization of debt issuance costs	71	0	0
Lending
Segment Reporting Information [Line Items]
Depreciation expense	93	103	194
Amortization of mortgage servicing rights			273
Amortization of debt discount	0	0	0
Amortization of debt issuance costs	0	0	0
Corporate Items and Other
Segment Reporting Information [Line Items]
Depreciation expense	29,893	22,498	20,895
Amortization of mortgage servicing rights			0
Amortization of debt discount	1,342	1,183	1,114
Amortization of debt issuance costs	$ 3,099	$ 2,921	$ 2,738