Mortgage Servicing Rights - Summary of Retention and Purchase of MSRs (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Mortgage Servicing Rights [Line Items]
MSR Fair Value	$ 7,387	$ 10,060	$ 64,725
MSRs
Mortgage Servicing Rights [Line Items]
MSR Fair Value	8,026	25,362	$ 95,281
Associated Principal	792,371	2,926,572
MSRs | Jumbo Loans
Mortgage Servicing Rights [Line Items]
MSR Fair Value	7,386	6,628
Associated Principal	686,263	966,705
MSRs | Jumbo Loans | Securitizations
Mortgage Servicing Rights [Line Items]
MSR Fair Value	7,123	6,451
Associated Principal	654,605	939,861
MSRs | Jumbo Loans | Loan Sales
Mortgage Servicing Rights [Line Items]
MSR Fair Value	263	177
Associated Principal	31,658	26,844
MSRs | Conforming Loans
Mortgage Servicing Rights [Line Items]
MSR Fair Value	640	18,734
Associated Principal	106,108	1,959,867
MSRs | Conforming Loans | Loan Sales
Mortgage Servicing Rights [Line Items]
MSR Fair Value	0	3,380
Associated Principal	0	316,290
MSRs | Conforming Loans | Loan Purchases
Mortgage Servicing Rights [Line Items]
MSR Fair Value	640	15,354
Associated Principal	$ 106,108	$ 1,643,577