Residential Whole Loans and Bridge Loans - Carrying Value Components (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Variable Interest Entity [Line Items]
Principal balance	$ 2,572,727
Residential Whole Loans, at Fair Value
Variable Interest Entity [Line Items]
Principal balance	989,143	$ 984,555
Unamortized premium	31,070	24,248
Unamortized discount	(1,337)	(1,799)
Amortized cost	1,018,876	1,007,004
Gross unrealized gains	14,190	9,282
Gross unrealized losses	(9,564)	(7,504)
Fair value	1,023,502	1,008,782
Residential Bridge Loans, at Fair Value
Variable Interest Entity [Line Items]
Principal balance	5,834	15,247
Unamortized premium	0	3
Unamortized discount	0	0
Amortized cost	5,834	15,250
Gross unrealized gains	78	5
Gross unrealized losses	(484)	(1,339)
Fair value	$ 5,428	13,916
Residential Bridge Loans, at Amortized Cost
Variable Interest Entity [Line Items]
Amortized cost		$ 1,100