Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Residential loan purchase commitments, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	$ (6)	$ (14,442)
Point HEI Noncontrolling Interest
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	0	(6,218)
Point Entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	1,194	9,628
Held-for-investment at fair value | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	156	(35,397)
Held-for-investment at fair value | Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	2,349	(4,981)
Held-for-investment at fair value | Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	8,759	2,940
Business purpose loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	12,239	(14,647)
Consolidated Agency multifamily loans | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	363	264
Single family rental loans | CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(8,810)	4,048
Trading Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	1,793	(1,401)
Available-for-sale securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(28)	0
HEIs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	3,433	1,185
Servicer Advance Investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(1,352)	(3,081)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(424)	3,526
Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(229)	(1,208)
Residential loan purchase commitments, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	353	2,050
Other investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	$ (94)	$ 0