Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	777,972	$ 810,238
Domestic servicing rights	16,524	16,917
Derivative assets	101,448	28,943
Total Assets	77,577,189	78,042,336
Total Liabilities	72,759,838	72,905,322
Derivative liabilities	5,118	8,740
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Derivative assets	101,448	28,943
Total Assets	62,835,695	63,908,829
Derivative liabilities	5,118	8,740
Total Liabilities	59,812,424	60,752,234
Fair value measurements on recurring basis | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	1,347,797	1,436,194
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	170,640	189,576
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	32,855	37,360
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	8,626	12,664
Fair value measurements on recurring basis | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	16,524	16,917
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total Assets	8,626	12,664
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	8,626	12,664
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	101,448	28,943
Total Assets	111,868	41,295
Derivative liabilities	5,118	8,740
Total Liabilities	58,305,987	58,214,842
Fair value measurements on recurring basis | Level II | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	10,420	12,352
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Total Assets	62,715,201	63,854,870
Total Liabilities	1,506,437	2,537,392
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	1,347,797	1,436,194
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	170,640	189,576
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	22,435	25,008
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	16,524	16,917
Primary beneficiary
Assets and liabilities measured at fair value
Total Assets	61,157,805	62,187,175
Total Liabilities	59,807,306	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | VIE Assets
Assets and liabilities measured at fair value
Total Assets	61,157,805	62,187,175
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	59,807,306	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	58,300,869	58,206,102
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Assets and liabilities measured at fair value
Total Assets	61,157,805	62,187,175
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	$ 1,506,437	$ 2,537,392