Mortgage Servicing Rights - Summary of Retention and Purchase of MSRs (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Mortgage Servicing Rights [Line Items]
MSR Fair Value	$ 10,060	$ 64,725	$ 48,000
MSRs
Mortgage Servicing Rights [Line Items]
MSR Fair Value	25,362	95,281	$ 95,550
Associated Principal	2,926,572	9,119,764
MSRs | Jumbo Loans
Mortgage Servicing Rights [Line Items]
MSR Fair Value	6,628	8,554
Associated Principal	966,705	915,882
MSRs | Jumbo Loans | Securitizations
Mortgage Servicing Rights [Line Items]
MSR Fair Value	6,451	8,202
Associated Principal	939,861	882,860
MSRs | Jumbo Loans | Loan Sales
Mortgage Servicing Rights [Line Items]
MSR Fair Value	177	352
Associated Principal	26,844	33,022
MSRs | Conforming Loans
Mortgage Servicing Rights [Line Items]
MSR Fair Value	18,734	86,727
Associated Principal	1,959,867	8,203,882
MSRs | Conforming Loans | Loan Sales
Mortgage Servicing Rights [Line Items]
MSR Fair Value	3,380	55,954
Associated Principal	316,290	5,251,537
MSRs | Conforming Loans | Loan Purchases
Mortgage Servicing Rights [Line Items]
MSR Fair Value	15,354	30,773
Associated Principal	$ 1,643,577	$ 2,952,345