Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019	Sep. 30, 2018
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income (Expense), Net	$ 431	$ 1,967	$ 2,342	$ 4,797
Gain on re-measurement of 5 Arches investment	0	0	2,441	0
Total Market Valuation Gains, Net	14,549	19,530	62,720	53,666
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	6,205	10,911	33,625	46,663
Mortgage Banking Activities, Net | Residential loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(6,623)	5,626	289	16,522
Mortgage Banking Activities, Net | Residential loan purchase and forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	12,943	1,610	41,142	(8,116)
Mortgage Banking Activities, Net | Single-family rental loans held-for-sale, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	1,283	(99)	4,200	(99)
Mortgage Banking Activities, Net | Single-family rental loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	564	(22)	1,273	(22)
Mortgage Banking Activities, Net | Residential bridge loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	1,010	0	2,108	0
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(2,972)	3,796	(15,387)	38,378
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	11,444	10,332	34,741	12,830
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(37,433)	21,867	(144,548)	82,391
Investment Fair Value Changes, Net | Residential loans held-for-investment, at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	7,667	(17,063)	71,323	(71,058)
Investment Fair Value Changes, Net | Single-family rental loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	22	0	22	0
Investment Fair Value Changes, Net | Residential bridge loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(742)	53	(1,363)	53
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	15,275	6,314	55,577	2,429
Investment Fair Value Changes, Net | Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	1,585	0	3,025	0
Investment Fair Value Changes, Net | Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(1,635)	0	(2,137)	0
Investment Fair Value Changes, Net | Shared home appreciation options
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	29	0	29	0
Investment Fair Value Changes, Net | REO
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(331)	0	(470)	0
Investment Fair Value Changes, Net | Legacy Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(407)	(248)	(904)	(976)
Investment Fair Value Changes, Net | Sequoia Choice
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	2,722	(943)	8,866	43
Investment Fair Value Changes, Net | Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	17,300	0	31,702	0
Investment Fair Value Changes, Net | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	7,445	511	13,810	511
Investment Fair Value Changes, Net | Risk-sharing investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(53)	(126)	(191)	(474)
Investment Fair Value Changes, Net | Impairments on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	0	(33)	0	(89)
Other Income (Expense), Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income (Expense), Net	(3,100)	(1,713)	(5,646)	(5,827)
Gain on re-measurement of 5 Arches investment	0	0	2,440	0
Other Income (Expense), Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income (Expense), Net	4,389	(890)	13,157	(7,151)
Other Income (Expense), Net | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Other Income (Expense), Net	$ (7,489)	$ (823)	$ (21,243)	$ 1,324