Residential Whole Loans and Bridge Loans Geographic Concentration (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Variable Interest Entity [Line Items]
Principal balance	$ 2,676,231
Residential Whole Loans
Variable Interest Entity [Line Items]
Principal balance	$ 1,325,443	$ 1,023,524
Geographic Concentration Risk | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	100.00%	100.00%
Principal balance	$ 1,325,443	$ 1,023,524
Geographic Concentration Risk | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	100.00%	100.00%
Principal balance	$ 37,196	$ 222,257
Geographic Concentration Risk | California | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	66.10%	67.10%
Principal balance	$ 875,738	$ 686,275
Geographic Concentration Risk | California | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	50.40%	53.90%
Principal balance	$ 18,763	$ 119,761
Geographic Concentration Risk | New York | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	16.20%	17.10%
Principal balance	$ 214,141	$ 175,390
Geographic Concentration Risk | New York | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	12.10%	9.50%
Principal balance	$ 4,518	$ 21,160
Geographic Concentration Risk | Georgia | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	3.40%	2.60%
Principal balance	$ 45,189	$ 26,918
Geographic Concentration Risk | Florida | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	2.80%	1.90%
Principal balance	$ 36,641	$ 19,942
Geographic Concentration Risk | Florida | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	8.90%	5.70%
Principal balance	$ 3,296	$ 12,672
Geographic Concentration Risk | New Jersey | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	2.30%
Principal balance	$ 30,450
Geographic Concentration Risk | New Jersey | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	3.80%	4.70%
Principal balance	$ 1,424	$ 10,419
Geographic Concentration Risk | Other | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	9.20%	9.20%
Principal balance	$ 123,284	$ 93,802
Geographic Concentration Risk | Other | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	11.70%	19.60%
Principal balance	$ 4,332	$ 43,534
Geographic Concentration Risk | Massachusetts | Residential Whole Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration		2.10%
Principal balance		$ 21,197
Geographic Concentration Risk | Washington | Residential Bridge Loans | Financing Receivables Total
Variable Interest Entity [Line Items]
Concentration	13.10%	6.60%
Principal balance	$ 4,863	$ 14,711