Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	$ (1,424)	$ 1,654	$ (3,233)	$ 1,911
Gain on re-measurement of 5 Arches investment	0	0	0	2,441
Total Market Valuation Gains (Losses), Net	146,830	16,732	(765,647)	48,172
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,443)	15,730	(38,446)	27,420
Mortgage Banking Activities, Net | Residential loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(2,014)	3,379	(15,494)	6,912
Mortgage Banking Activities, Net | Residential loan purchase and forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	621	16,888	22,056	28,199
Mortgage Banking Activities, Net | Single-family rental loans held-for-sale, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	1,210	1,313	12,677	2,917
Mortgage Banking Activities, Net | Single-family rental loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	569	341	709
Mortgage Banking Activities, Net | Residential bridge loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,260)	1,012	(5,194)	1,098
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	(7,431)	(52,832)	(12,415)
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	152,228	3,138	(718,604)	23,297
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	(64,740)	(59,142)	(107,115)
Investment Fair Value Changes, Net | Residential loans held-for-investment, at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	104	35,548	(93,532)	63,656
Investment Fair Value Changes, Net | Single-family rental loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	2,222	0	(20,806)	0
Investment Fair Value Changes, Net | Residential bridge loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	21,774	(318)	(16,828)	(621)
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	42,246	18,442	(221,079)	40,302
Investment Fair Value Changes, Net | Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(136)	432	(6,198)	1,440
Investment Fair Value Changes, Net | Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	2,971	(65)	(6,523)	(502)
Investment Fair Value Changes, Net | Legacy Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(230)	(123)	(621)	(497)
Investment Fair Value Changes, Net | Sequoia Choice
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	39,753	2,879	(29,916)	6,144
Investment Fair Value Changes, Net | Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	26,867	8,037	(115,295)	14,402
Investment Fair Value Changes, Net | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	1,599	3,246	(84,910)	6,365
Investment Fair Value Changes, Net | CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	17,125	0	(50,721)	0
Investment Fair Value Changes, Net | Other investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(2,121)	(200)	(11,562)	(277)
Investment Fair Value Changes, Net | Credit recoveries (losses) on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	54	0	(1,471)	0
Other Income
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	(3,955)	(2,136)	(8,597)	(2,545)
Gain on re-measurement of 5 Arches investment	0	0	0	2,440
Other Income | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	0	6,517	13,966	8,768
Other Income | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	$ (3,955)	$ (8,653)	$ (22,563)	$ (13,753)