Borrowings - Assets Sold Under Agreement to Repurchase (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2019	Mar. 31, 2018	Dec. 31, 2018
During the period:
Total interest expense	$ 8,635	$ 6,732
Carrying value:
Unpaid principal balance	2,152,588
Unamortized debt issuance costs and premiums	(650)		$ (1,341)
Total loans sold under agreements to repurchase	2,151,938		1,933,859
Servicing advances	147,435		162,895
Assets sold under agreements to repurchase
During the period:
Average balance of mortgage loans sold under agreements to repurchase	$ 1,437,957	$ 1,643,443
Weighted-average interest rate (as a percent)	4.47%	3.59%
Total interest expense	$ 8,635	$ 6,732
Maximum daily amount outstanding	2,152,588	2,380,121
Carrying value:
Unpaid principal balance	2,152,588		1,935,200
Unamortized debt issuance costs and premiums	(650)		(1,341)
Total loans sold under agreements to repurchase	$ 2,151,938		$ 1,933,859
Weighted average interest rate (as a percent)	4.35%		4.22%
Available borrowing capacity committed	$ 625,413		$ 695,767
Available borrowing capacity uncommitted	2,206,999		2,354,033
Available borrowing capacity	2,832,412		3,049,800
Margin deposits placed with counterparties	3,750		3,750
Amortization of premium	7,400	8,000
Amount of Master Repurchase Agreement incentives to refinance include in interest expense	9,300	$ 10,200
Assets sold under agreements to repurchase | Mortgage Loans held for sale
Carrying value:
Fair value of assets pledged to secure	2,065,151		1,923,857
Assets sold under agreements to repurchase | Mortgage servicing rights
Carrying value:
Fair value of assets pledged to secure	2,574,228		2,807,333
Assets sold under agreements to repurchase | Servicing advances
Carrying value:
Fair value of assets pledged to secure	147,435		162,895
Assets sold under agreements to repurchase | Financing receivable
Carrying value:
Fair value of assets pledged to secure	$ 125,929		$ 131,025