Borrowings - Assets Sold Under Agreement to Repurchase (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019
During the period:
Total interest expense	$ 25,684	$ 8,635
Carrying value:
Unpaid principal balance	4,446,795		$ 4,141,680
Unamortized debt issuance costs and premiums	(2,250)		(627)
Unamortized debt issuance costs	(2,250)		(627)
Total loans sold under agreements to repurchase	$ 4,444,545		$ 4,141,053
Weighted average interest rate (as a percent)	2.54%		3.29%
Available borrowing capacity committed			$ 125,810
Available borrowing capacity uncommitted	$ 1,403,205		782,510
Available borrowing capacity	1,403,205		908,320
Servicing advances	182,531		207,460
Margin deposits placed with counterparties	5,000		5,000
Loans held for sale
Carrying value:
Fair value of assets pledged to secure	4,937,094		4,322,789
Mortgage servicing rights
Carrying value:
Fair value of assets pledged to secure	2,151,501		2,902,721
Servicing advances
Carrying value:
Fair value of assets pledged to secure	182,531		207,460
Financing receivable
Carrying value:
Fair value of assets pledged to secure	99,766		$ 107,512
Assets sold under agreements to repurchase
During the period:
Average balance of assets sold under agreements to repurchase	$ 3,139,328	$ 1,437,957
Weighted-average interest rate (as a percent)	3.07%	4.47%
Total interest expense	$ 25,684	$ 8,635
Maximum daily amount outstanding	4,446,795	2,152,588
Carrying value:
Amortization of premium	$ 1,600	7,400
Amount of Master Repurchase Agreement incentives to refinance include in interest expense		$ 9,300