Fair Value of Financial Instruments - Additional Information (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Securities
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	$ 108,000	$ 333,000
Residential Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	4,370,000	6,990,000
Business purpose residential loans, held-for-sale, at fair value
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	1,400,000	4,020,000
Multifamily securities
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	$ 0	1,430,000
Real estate securities | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Dealer marks of securities	75.00%
Percentage of carrying value for which dealer quotes were received on securities	88.00%
Percentage difference of internal valuation than dealer marks	3.00%
Servicer advance investments
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	$ 179,000	70,000
Excess MSRs
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	11,000	8,000
Shared Home Appreciation Options
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value option elected aggregate carrying amount, asset	$ 4,000	$ 43,000