Residential Whole Loans and Bridge Loans - Geographic Concentration (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020
Variable Interest Entity [Line Items]
Principal balance	$ 2,572,727
Residential Whole Loans
Variable Interest Entity [Line Items]
Principal balance	$ 989,143	$ 984,555
Geographic Concentration Risk | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	100.00%	100.00%
Principal balance	$ 989,143	$ 984,555
Geographic Concentration Risk | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	100.00%	100.00%
Principal balance	$ 5,834	$ 15,247
Geographic Concentration Risk | CALIFORNIA | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	73.90%	65.80%
Principal balance	$ 730,771	$ 647,877
Geographic Concentration Risk | CALIFORNIA | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	30.10%	37.50%
Principal balance	$ 1,754	$ 5,713
Geographic Concentration Risk | NEW YORK | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	11.60%	17.70%
Principal balance	$ 114,625	$ 173,788
Geographic Concentration Risk | NEW YORK | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	45.10%	17.30%
Principal balance	$ 2,631	$ 2,632
Geographic Concentration Risk | FLORIDA | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	2.70%	2.80%
Principal balance	$ 26,293	$ 27,274
Geographic Concentration Risk | FLORIDA | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	19.30%	12.90%
Principal balance	$ 1,125	$ 1,969
Geographic Concentration Risk | WASHINGTON | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration		16.10%
Principal balance		$ 2,461
Geographic Concentration Risk | TEXAS | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	1.90%
Principal balance	$ 19,062
Geographic Concentration Risk | NEW JERSEY | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration		2.50%
Principal balance		$ 24,704
Geographic Concentration Risk | NEW JERSEY | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	3.70%
Principal balance	$ 219
Geographic Concentration Risk | PENNSYLVANIA | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	1.80%
Principal balance	$ 105
Geographic Concentration Risk | GEORGIA | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	2.50%	3.40%
Principal balance	$ 25,106	$ 33,577
Geographic Concentration Risk | CONNECTICUT | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration		5.70%
Principal balance		$ 872
Geographic Concentration Risk | Other States | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	7.40%	7.80%
Principal balance	$ 73,286	$ 77,335
Geographic Concentration Risk | Other States | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
State Concentration	0.00%	10.50%
Principal balance	$ 0	$ 1,600