Residential Whole-Loans and Bridge Loans - Investment Portfolio (Details) $ in Thousands	3 Months Ended	6 Months Ended	12 Months Ended
	Mar. 31, 2020 USD ($) loan score	Jun. 30, 2019 loan	Dec. 31, 2019 USD ($) loan score
Variable Interest Entity [Line Items]
Contractual  Maturity  (years)			5 months 18 days
Fair value of residential whole-loans pledged as collateral	$ 1,309,795		$ 1,375,860
VIE
Variable Interest Entity [Line Items]
Fair value of residential whole-loans pledged as collateral	$ 1,309,795		$ 1,375,860
3.01 – 4.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	52		53
Principal  Balance	$ 17,940		$ 17,284
Original LTV	62.70%		61.70%
Contractual  Maturity  (years)	28 years 1 month 6 days		28 years
Coupon  Rate	3.90%		3.90%
Original FICO Score | score	734		736
Expected  Life (years)(2)	2 years 9 months 18 days		2 years 4 months 24 days
4.01– 5.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	1,716		1,689
Principal  Balance	$ 588,582		$ 557,144
Original LTV	61.80%		61.40%
Contractual  Maturity  (years)	28 years 6 months		28 years 6 months
Coupon  Rate	4.80%		4.80%
Original FICO Score | score	746		744
Expected  Life (years)(2)	2 years 3 months 18 days		2 years 9 months 18 days
5.01 – 6.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	1,707		1,682
Principal  Balance	$ 723,264		$ 713,397
Original LTV	62.20%		62.00%
Contractual  Maturity  (years)	28 years 1 month 6 days		28 years 3 months 18 days
Coupon  Rate	5.40%		5.40%
Original FICO Score | score	736		736
Expected  Life (years)(2)	2 years 7 months 6 days		3 years
6.01 – 7.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	66		103
Principal  Balance	$ 22,477		$ 37,102
Original LTV	58.20%		54.10%
Contractual  Maturity  (years)	26 years 9 months 18 days		25 years 3 months 18 days
Coupon  Rate	6.20%		6.20%
Original FICO Score | score	724		727
Expected  Life (years)(2)	3 years 6 months		3 years 9 months 18 days
7.01 - 8.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	2		2
Principal  Balance	$ 515		$ 516
Original LTV	73.20%		73.20%
Contractual  Maturity  (years)	28 years 4 months 24 days		28 years 7 months 6 days
Coupon  Rate	7.10%		7.10%
Original FICO Score | score	753		753
Expected  Life (years)(2)	4 years		4 years 8 months 12 days
Residential  Whole Loans
Variable Interest Entity [Line Items]
Number of Loans | loan	3,543		3,529
Principal  Balance	$ 1,352,778		$ 1,325,443
Original LTV	62.00%		61.50%
Contractual  Maturity  (years)	28 years 2 months 12 days		28 years 3 months 18 days
Coupon  Rate	5.10%		5.20%
Original FICO Score | score	740		739
Expected  Life (years)(2)	2 years 6 months		3 years
Residential  Whole Loans | VIE
Variable Interest Entity [Line Items]
Number of Loans | loan	3,534
Residential portfolio segment with no FICO score
Variable Interest Entity [Line Items]
Number of Loans | loan	282	286
Principal  Balance	$ 94,400		$ 94,600
7.01 – 9.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	26
Principal  Balance	$ 17,823
Original LTV	67.40%
Coupon  Rate	8.40%
9.01 – 11.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	22		36
Principal  Balance	$ 8,711		$ 22,409
Original LTV	75.20%		70.20%
Contractual  Maturity  (years)	4 months 3 days		5 months 24 days
Coupon  Rate	10.20%		8.40%
11.01 – 13.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	6		28
Principal  Balance	$ 2,167		$ 9,972
Original LTV	64.30%		74.00%
Contractual  Maturity  (years)	4 months 9 days		5 months 18 days
Coupon  Rate	11.70%		10.10%
13.01 – 15.00%
Variable Interest Entity [Line Items]
Number of Loans | loan			9
Principal  Balance			$ 2,741
Original LTV			63.10%
Contractual  Maturity  (years)	2 months		2 months
Coupon  Rate			11.70%
15.01 – 17.00%
Variable Interest Entity [Line Items]
Number of Loans | loan			1
Principal  Balance			$ 1,125
Original LTV			75.00%
Contractual  Maturity  (years)	0 months		0 months
Coupon  Rate			13.50%
17.01 – 19.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	2		2
Principal  Balance	$ 949		$ 949
Original LTV	75.00%		75.00%
Contractual  Maturity  (years)	4 months 3 days		0 months
Coupon  Rate	18.00%		18.00%
Residential Bridge Loans | VIE
Variable Interest Entity [Line Items]
Number of Loans | loan	56		76
Principal  Balance	$ 29,650		$ 37,196
Original LTV	69.70%		71.00%
Coupon  Rate	9.50%		9.50%
Residential Whole-Loan And Residential Bridge Loan | VIE
Variable Interest Entity [Line Items]
Fair value of residential whole-loans pledged as collateral	$ 1,309,795		$ 1,375,860
Minimum | 3.01 – 4.00%
Variable Interest Entity [Line Items]
Coupon  Rate	3.01%	3.01%
Minimum | 4.01– 5.00%
Variable Interest Entity [Line Items]
Coupon  Rate	4.01%	4.01%
Minimum | 5.01 – 6.00%
Variable Interest Entity [Line Items]
Coupon  Rate	5.01%	5.01%
Minimum | 6.01 – 7.00%
Variable Interest Entity [Line Items]
Coupon  Rate	6.01%	6.01%
Minimum | 7.01 - 8.00%
Variable Interest Entity [Line Items]
Coupon  Rate	7.01%	7.01%
Minimum | 7.01 – 9.00%
Variable Interest Entity [Line Items]
Coupon  Rate	701.00%
Minimum | 9.01 – 11.00%
Variable Interest Entity [Line Items]
Coupon  Rate	901.00%		901.00%
Minimum | 11.01 – 13.00%
Variable Interest Entity [Line Items]
Coupon  Rate	1101.00%		1101.00%
Minimum | 13.01 – 15.00%
Variable Interest Entity [Line Items]
Coupon  Rate			1301.00%
Minimum | 15.01 – 17.00%
Variable Interest Entity [Line Items]
Coupon  Rate			1501.00%
Minimum | 17.01 – 19.00%
Variable Interest Entity [Line Items]
Coupon  Rate	1701.00%		1701.00%
Maximum | 3.01 – 4.00%
Variable Interest Entity [Line Items]
Coupon  Rate	4.00%	4.00%
Maximum | 4.01– 5.00%
Variable Interest Entity [Line Items]
Coupon  Rate	5.00%	5.00%
Maximum | 5.01 – 6.00%
Variable Interest Entity [Line Items]
Coupon  Rate	6.00%	6.00%
Maximum | 6.01 – 7.00%
Variable Interest Entity [Line Items]
Coupon  Rate	7.00%	7.00%
Maximum | 7.01 - 8.00%
Variable Interest Entity [Line Items]
Coupon  Rate	8.00%	8.00%
Maximum | 7.01 – 9.00%
Variable Interest Entity [Line Items]
Coupon  Rate	900.00%
Maximum | 9.01 – 11.00%
Variable Interest Entity [Line Items]
Coupon  Rate	1100.00%		1100.00%
Maximum | 11.01 – 13.00%
Variable Interest Entity [Line Items]
Coupon  Rate	1300.00%		1300.00%
Maximum | 13.01 – 15.00%
Variable Interest Entity [Line Items]
Coupon  Rate			1500.00%
Maximum | 15.01 – 17.00%
Variable Interest Entity [Line Items]
Coupon  Rate			1700.00%
Maximum | 17.01 – 19.00%
Variable Interest Entity [Line Items]
Coupon  Rate	1900.00%		1900.00%