Investments in Servicing Related Assets, Summary (Details) - USD ($) $ in Thousands		3 Months Ended	12 Months Ended
		Mar. 31, 2017	Dec. 31, 2016
Investment [Line Items]
Unpaid principal balance		$ 7,591,712	$ 9,315,323
Cost basis		64,386	56,097
Carrying value	[1]	$ 76,698	$ 61,263
Weighted average coupon		3.55%	3.26%
Weighted average maturity	[2]	26 years 4 months 24 days	25 years 4 months 24 days
Changes in fair value recorded in other income (loss)	[3]	$ 12,312	$ (3,036)
Excess MSR Pool 2 [Member]
Investment [Line Items]
Unpaid principal balance			6,053,142
Cost basis	[4]		19,754
Carrying value	[1]		$ 28,526
Weighted average coupon			2.96%
Weighted average maturity	[2]		26 years 3 months 18 days
Changes in fair value recorded in other income (loss)	[3]		$ (493)
Excess MSR Pool 2 - Recapture Agreement [Member]
Investment [Line Items]
Unpaid principal balance			0
Cost basis	[4]		1,187
Carrying value	[1]		$ 866
Weighted average maturity	[2]		0 years
Changes in fair value recorded in other income (loss)	[3]		$ 742
Mortgage Service Right Conventional [Member]
Investment [Line Items]
Unpaid principal balance		3,170,673	3,262,181
Cost basis	[5]	31,871	35,156
Carrying value	[1]	$ 31,088	$ 31,871
Weighted average coupon		3.81%	3.81%
Weighted average maturity	[2]	23 years 6 months	23 years 8 months 12 days
Changes in fair value recorded in other income (loss)	[3]	$ (783)	$ (3,285)
Mortgage Service Right Government [Member]
Investment [Line Items]
Unpaid principal balance		4,421,039
Cost basis	[5]	32,515
Carrying value	[1]	$ 45,610
Weighted average coupon		3.36%
Weighted average maturity	[2]	28 years 6 months
Changes in fair value recorded in other income (loss)	[3]	$ 13,095

[1] Carrying value represents the fair value of the pools or recapture agreements, as applicable (see Note 9).

[2] The weighted average maturity represents the weighted average expected timing of the receipt of cash flows of each investment.

[3] The portion of the change in fair value of the recapture agreement relating to loans recaptured as of March 31, 2017 and December 31, 2016 is reflected in the respective pool.

[4] The amortized cost basis of the recapture agreements is determined based on the relative fair values of the recapture agreements and related Excess MSRs at the time they were acquired.

[5] MSR cost basis consists of the carrying value of the prior period, adjusted for any purchases, sales and principal paydowns.