Investments in Servicing Related Assets, Summary (Details) - USD ($) $ in Thousands		12 Months Ended
		Dec. 31, 2019	Dec. 31, 2018
Servicing Asset [Abstract]
Unpaid principal balance		$ 29,068,126	$ 24,846,989
Cost basis		397,883	298,480
Carrying value	[1]	$ 291,111	$ 294,907
Weighted average coupon		4.18%	4.23%
Weighted average maturity	[2]	26 years 8 months 12 days	27 years 2 months 12 days
Changes in fair value recorded in other income (loss)		$ (106,772)	$ (3,573)
Mortgage Service Right Conventional [Member]
Servicing Asset [Abstract]
Unpaid principal balance		26,142,780	21,366,980
Cost basis	[3]	357,667	258,070
Carrying value	[1]	$ 263,357	$ 254,691
Weighted average coupon		4.27%	4.37%
Weighted average maturity	[2]	26 years 9 months 18 days	27 years 3 months 18 days
Changes in fair value recorded in other income (loss)		$ (94,310)	$ (3,379)
Mortgage Service Right Government [Member]
Servicing Asset [Abstract]
Unpaid principal balance		2,925,346	3,480,009
Cost basis	[3]	40,216	40,410
Carrying value	[1]	$ 27,754	$ 40,216
Weighted average coupon		3.37%	3.37%
Weighted average maturity	[2]	25 years 9 months 18 days	26 years 9 months 18 days
Changes in fair value recorded in other income (loss)		$ (12,462)	$ (194)

[1] Carrying value approximates the fair value of the pools (see Note 9).

[2] The weighted average maturity represents the weighted average expected timing of the receipt of cash flows of each investment.

[3] MSR cost basis consists of the carrying value of the prior period, adjusted for any purchases, sales and principal paydowns of the underlying mortgage loans.