Borrowings - Assets Sold Under Agreement to Repurchase (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Carrying value:
Unpaid principal balance	$ 4,141,680
Total loans sold under agreements to repurchase	4,141,053	$ 1,933,859
Fair value of assets pledged to secure	4,322,789	1,923,857
Servicing advances	207,460	162,895
Assets sold under agreements to repurchase
During the period:
Average balance of assets sold under agreements to repurchase	$ 2,185,830	$ 1,626,729	$ 1,829,257
Weighted-average interest rate (as a percent)	3.74%	3.87%	3.18%
Total interest expense	$ 74,215	$ 22,463	$ 60,286
Maximum daily amount outstanding	4,141,680	2,380,121	3,022,656
Carrying value:
Unpaid principal balance	4,141,680	1,935,200
Unamortized debt issuance costs and premiums	(627)	(1,341)
Total loans sold under agreements to repurchase	$ 4,141,053	$ 1,933,859
Weighted average interest rate (as a percent)	3.29%	4.22%
Available borrowing capacity committed	$ 125,810	$ 695,767
Available borrowing capacity uncommitted	782,510	2,354,033
Available borrowing capacity	908,320	3,049,800
Margin deposits placed with counterparties	5,000	3,750
Amortization of premium	7,500	40,500	1,300
Amount of Master Repurchase Agreement incentives to refinance include in interest expense	14,700	48,100	$ 9,200
Beneficial interest in mortgage servicing rights	2,800,000
Assets sold under agreements to repurchase | Loans held for sale
Carrying value:
Fair value of assets pledged to secure	4,322,789	1,923,857
Assets sold under agreements to repurchase | Servicing advances
Carrying value:
Servicing advances	207,460	162,895
Assets sold under agreements to repurchase | Financing receivable | PMT
Carrying value:
Fair value of assets pledged to secure	107,512	131,025
Assets sold under agreements to repurchase | Mortgage servicing rights
Carrying value:
Fair value of assets pledged to secure	$ 2,902,721	$ 2,807,333