Residential Whole-Loans and Bridge Loans - Collateral Securing (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2020	Dec. 31, 2019
Residential  Whole Loans
Variable Interest Entity [Line Items]
Principal  Balance	$ 1,352,778	$ 1,325,443
Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	100.00%	100.00%
Principal  Balance	$ 1,352,778	$ 1,325,443
Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	100.00%	100.00%
Principal  Balance	$ 29,650	$ 37,196
California | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	66.50%	66.10%
Principal  Balance	$ 899,344	$ 875,738
California | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	49.20%	50.40%
Principal  Balance	$ 14,598	$ 18,763
New York | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	15.90%	16.20%
Principal  Balance	$ 214,658	$ 214,141
New York | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	14.60%	13.10%
Principal  Balance	$ 4,323	$ 4,863
Georgia | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	3.10%	3.40%
Principal  Balance	$ 42,506	$ 45,189
Florida | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	2.80%	2.80%
Principal  Balance	$ 38,154	$ 36,641
Florida | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	10.50%	8.90%
Principal  Balance	$ 3,128	$ 3,296
New Jersey | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	2.20%	2.30%
Principal  Balance	$ 29,367	$ 30,450
New Jersey | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration		3.80%
Principal  Balance		$ 1,424
Other | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
Concentration	9.50%	9.20%
Principal  Balance	$ 128,749	$ 123,284
Other | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	9.70%	11.70%
Principal  Balance	$ 2,849	$ 4,332
Washington | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	12.60%	12.10%
Principal  Balance	$ 3,748	$ 4,518
New Jersey | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
Concentration	3.40%
Principal  Balance	$ 1,004