Investments in Servicing Related Assets, Summary (Details) - USD ($) $ in Thousands		12 Months Ended
		Dec. 31, 2018	Dec. 31, 2017
Servicing Asset [Abstract]
Unpaid principal balance		$ 24,846,989	$ 11,710,651
Cost basis		298,480	113,647
Carrying value	[1]	$ 294,907	$ 122,806
Weighted average coupon		4.23%	3.71%
Weighted average maturity	[2]	27 years 2 months 12 days	26 years 1 month 6 days
Changes in fair value recorded in other income (loss)		$ (3,573)	$ 9,159
Mortgage Service Right Conventional [Member]
Servicing Asset [Abstract]
Unpaid principal balance		21,366,980	7,724,397
Cost basis	[3]	258,070	81,499
Carrying value	[1]	$ 254,691	$ 82,150
Weighted average coupon		4.37%	3.89%
Weighted average maturity	[2]	27 years 3 months 18 days	25 years 2 months 12 days
Changes in fair value recorded in other income (loss)		$ (3,379)	$ 651
Mortgage Service Right Government [Member]
Servicing Asset [Abstract]
Unpaid principal balance		3,480,009	3,986,254
Cost basis	[3]	40,410	32,148
Carrying value	[1]	$ 40,216	$ 40,656
Weighted average coupon		3.37%	3.36%
Weighted average maturity	[2]	26 years 9 months 18 days	27 years 9 months 18 days
Changes in fair value recorded in other income (loss)		$ (194)	$ 8,508

[1] Carrying value represents the fair value of the pools (see Note 9).

[2] The weighted average maturity represents the weighted average expected timing of the receipt of cash flows of each investment.

[3] MSR cost basis consists of the carrying value of the prior period, adjusted for any purchases, sales and principal paydowns of the underlying mortgage loans.