Investments in Servicing Related Assets, Summary (Details) - USD ($) $ in Thousands		9 Months Ended	12 Months Ended
		Sep. 30, 2019	Dec. 31, 2018
Servicing Asset [Abstract]
Unpaid principal balance		$ 27,864,210	$ 24,846,989
Cost basis		364,576	298,480
Carrying value	[1]	$ 255,845	$ 294,907
Weighted average coupon		4.25%	4.23%
Weighted average maturity	[2]	26 years 9 months 18 days	27 years 2 months 12 days
Changes in fair value recorded in other income (loss)		$ (108,731)	$ (3,573)
Mortgage Service Right Conventional [Member]
Servicing Asset [Abstract]
Unpaid principal balance		24,794,818	21,366,980
Cost basis	[3]	324,360	258,070
Carrying value	[1]	$ 227,522	$ 254,691
Weighted average coupon		4.36%	4.37%
Weighted average maturity	[2]	26 years 10 months 24 days	27 years 3 months 18 days
Changes in fair value recorded in other income (loss)		$ (96,838)	$ (3,379)
Mortgage Service Right Government [Member]
Servicing Asset [Abstract]
Unpaid principal balance		3,069,392	3,480,009
Cost basis	[3]	40,216	40,410
Carrying value	[1]	$ 28,323	$ 40,216
Weighted average coupon		3.37%	3.37%
Weighted average maturity	[2]	26 years 1 month 6 days	26 years 9 months 18 days
Changes in fair value recorded in other income (loss)		$ (11,893)	$ (194)

[1] Carrying value represents the fair value of the pools (see Note 9).

[2] The weighted average maturity represents the weighted average expected timing of the receipt of cash flows of each investment.

[3] MSR cost basis consists of the carrying value of the prior period, adjusted for any purchases, sales and principal paydowns of the underlying mortgage loans.