Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	$ (2,362)	$ 431	$ (5,595)	$ 2,342
Gain on re-measurement of 5 Arches investment	0	0	0	2,441
Total Market Valuation Gains (Losses), Net	158,883	14,549	(606,764)	62,720
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	56,619	6,205	18,173	33,625
Mortgage Banking Activities, Net | Residential loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(478)	(6,623)	(15,972)	289
Mortgage Banking Activities, Net | Residential loan purchase and forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	13,067	12,943	35,123	41,142
Mortgage Banking Activities, Net | Single-family rental loans held-for-sale, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	43,191	1,283	55,868	4,200
Mortgage Banking Activities, Net | Single-family rental loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	564	341	1,273
Mortgage Banking Activities, Net | Residential bridge loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	938	1,010	(4,256)	2,108
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(99)	(2,972)	(52,931)	(15,387)
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	107,047	11,444	(611,557)	34,741
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	(37,433)	(59,142)	(144,548)
Investment Fair Value Changes, Net | Residential loans held-for-investment, at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	218	7,667	(93,314)	71,323
Investment Fair Value Changes, Net | Single-family rental loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	22	(20,806)	22
Investment Fair Value Changes, Net | Residential bridge loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	6,812	(742)	(10,016)	(1,363)
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(3,600)	15,275	(224,679)	55,577
Investment Fair Value Changes, Net | Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	26	1,585	(6,172)	3,025
Investment Fair Value Changes, Net | Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,127)	(1,635)	(7,650)	(2,137)
Investment Fair Value Changes, Net | Legacy Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(81)	(407)	(702)	(904)
Investment Fair Value Changes, Net | Sequoia Choice
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	7,851	2,722	(22,065)	8,866
Investment Fair Value Changes, Net | Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	82,214	17,300	(33,081)	31,702
Investment Fair Value Changes, Net | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	2,166	7,445	(82,744)	13,810
Investment Fair Value Changes, Net | CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	9,673	0	(41,048)	0
Investment Fair Value Changes, Net | Other investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	2,451	(355)	(9,111)	(632)
Investment Fair Value Changes, Net | Credit recoveries (losses) on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	444	0	(1,027)	0
Other Income
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	(4,783)	(3,100)	(13,380)	(5,646)
Gain on re-measurement of 5 Arches investment	0	0	0	2,440
Other Income | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	0	4,389	13,966	13,157
Other Income | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income	$ (4,783)	$ (7,489)	$ (27,346)	$ (21,243)