Residential Whole Loans and Bridge Loans Investment Portfolio (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($) loan item score	Dec. 31, 2018 USD ($) loan item score
Variable Interest Entity [Line Items]
Principal balance | $	$ 2,676,231
3.01 - 4.00%
Variable Interest Entity [Line Items]
Number of Loans	53	66
Principal balance | $	$ 17,284	$ 22,046
Original LTV	61.70%	61.60%
Original FICO Score | item	736	738
Expected Life (in years)	2 years 4 months 24 days	6 years 6 months
Contractual Maturity (years)	28 years	29 years
Coupon Rate	3.90%	3.90%
4.01 - 5.00%
Variable Interest Entity [Line Items]
Number of Loans	1,689	1,395
Principal balance | $	$ 557,144	$ 490,073
Original LTV	61.40%	62.30%
Original FICO Score | item	744	739
Expected Life (in years)	2 years 9 months 18 days	3 years
Contractual Maturity (years)	28 years 6 months	29 years
Coupon Rate	4.80%	4.80%
5.01 - 6.00%
Variable Interest Entity [Line Items]
Number of Loans	1,682	1,283
Principal balance | $	$ 713,397	$ 496,722
Original LTV	62.00%	62.70%
Original FICO Score | item	736	727
Expected Life (in years)	3 years	2 years 6 months
Contractual Maturity (years)	28 years 3 months 18 days	28 years 6 months
Coupon Rate	5.40%	5.40%
6.01 - 7.00%
Variable Interest Entity [Line Items]
Number of Loans	103	37
Principal balance | $	$ 37,102	$ 14,589
Original LTV	54.10%	59.50%
Original FICO Score | item	727	731
Expected Life (in years)	3 years 9 months 18 days	1 year 6 months
Contractual Maturity (years)	25 years 3 months 18 days	24 years 9 months 18 days
Coupon Rate	6.20%	6.20%
7.01 - 8.00%
Variable Interest Entity [Line Items]
Number of Loans	2	1
Principal balance | $	$ 516	$ 94
Original LTV	73.20%	70.00%
Original FICO Score | score	753	689
Expected Life (in years)	4 years 8 months 12 days	1 year 9 months 18 days
Contractual Maturity (years)	28 years 7 months 6 days	29 years 1 month 6 days
Coupon Rate	7.10%	8.00%
Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	3,529	2,782
Principal balance | $	$ 1,325,443	$ 1,023,524
Original LTV	61.50%	62.40%
Original FICO Score | item	739	733
Expected Life (in years)	3 years	2 years 9 months 18 days
Contractual Maturity (years)	28 years 3 months 18 days	28 years 8 months 12 days
Coupon Rate	5.20%	5.10%
Residential portfolio segment with no FICO score
Variable Interest Entity [Line Items]
Number of Loans	286	274
Principal balance | $	$ 94,600	$ 93,200
5.01 - 7.00%
Variable Interest Entity [Line Items]
Number of Loans		8
Principal balance | $		$ 3,169
Original LTV		60.40%
Contractual Maturity (years)		1 year 1 month 6 days
Coupon Rate		6.70%
7.01 – 9.00%
Variable Interest Entity [Line Items]
Number of Loans	36	275
Principal balance | $	$ 22,409	$ 140,675
Original LTV	70.20%	72.60%
Contractual Maturity (years)	5 years 9 months 18 days	5 years 10 months 24 days
Coupon Rate	8.40%	8.30%
9.01 – 11.00%
Variable Interest Entity [Line Items]
Number of Loans		186
Principal balance | $		$ 63,954
Original LTV		73.80%
Contractual Maturity (years)		4 years 4 months 24 days
Coupon Rate		9.90%
11.01 – 13.00%
Variable Interest Entity [Line Items]
Number of Loans	28	39
Principal balance | $	$ 9,972	$ 11,017
Original LTV	74.00%	71.30%
Contractual Maturity (years)	5 years 7 months 6 days	4 years 7 months 6 days
Coupon Rate	10.10%	11.80%
13.01 – 15.00%
Variable Interest Entity [Line Items]
Number of Loans		1
Principal balance | $		$ 88
Original LTV		65.00%
Contractual Maturity (years)		4 years
Coupon Rate		14.00%
17.01 - 19.00%
Variable Interest Entity [Line Items]
Number of Loans	9	11
Principal balance | $	$ 2,741	$ 3,354
Original LTV	63.10%	73.70%
Contractual Maturity (years)	2 years	2 years 3 months 18 days
Coupon Rate	11.70%	18.00%
13.01 - 14.00%
Variable Interest Entity [Line Items]
Number of Loans	1
Principal balance | $	$ 1,125
Original LTV	75.00%
Contractual Maturity (years)	0 years
Coupon Rate	13.50%
17.01 – 19.00%
Variable Interest Entity [Line Items]
Number of Loans	2
Principal balance | $	$ 949
Original LTV	75.00%
Contractual Maturity (years)	0 years
Coupon Rate	18.00%
VIE | Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	3,520
VIE | Residential Bridge Loans
Variable Interest Entity [Line Items]
Number of Loans	76	520
Principal balance | $	$ 37,196	$ 222,257
Original LTV	71.00%	72.70%
Contractual Maturity (years)	5 years 7 months 6 days	5 years 3 months 18 days
Coupon Rate	9.50%	9.10%
Minimum | 3.01 - 4.00%
Variable Interest Entity [Line Items]
Coupon Rate	3.01%	3.01%
Minimum | 4.01 - 5.00%
Variable Interest Entity [Line Items]
Coupon Rate	4.01%	4.01%
Minimum | 5.01 - 6.00%
Variable Interest Entity [Line Items]
Coupon Rate	5.01%	5.01%
Minimum | 6.01 - 7.00%
Variable Interest Entity [Line Items]
Coupon Rate	6.01%	6.01%
Minimum | 5.01 - 7.00%
Variable Interest Entity [Line Items]
Coupon Rate		5.01%
Minimum | 7.01 – 9.00%
Variable Interest Entity [Line Items]
Coupon Rate	7.01%	7.01%
Minimum | 11.01 – 13.00%
Variable Interest Entity [Line Items]
Coupon Rate	9.01%	9.01%
Minimum | 17.01 - 19.00%
Variable Interest Entity [Line Items]
Coupon Rate	11.01%	11.01%
Minimum | 13.01 - 14.00%
Variable Interest Entity [Line Items]
Coupon Rate	13.01%	13.01%
Minimum | 17.01 – 19.00%
Variable Interest Entity [Line Items]
Coupon Rate	17.01%	17.01%
Minimum | Residential Portfolio Segment Current Coupon Rate Five
Variable Interest Entity [Line Items]
Coupon Rate	7.01%	7.01%
Maximum | 3.01 - 4.00%
Variable Interest Entity [Line Items]
Coupon Rate	4.00%	4.00%
Maximum | 4.01 - 5.00%
Variable Interest Entity [Line Items]
Coupon Rate	5.00%	5.00%
Maximum | 5.01 - 6.00%
Variable Interest Entity [Line Items]
Coupon Rate	6.00%	6.00%
Maximum | 6.01 - 7.00%
Variable Interest Entity [Line Items]
Coupon Rate	7.00%	7.00%
Maximum | 5.01 - 7.00%
Variable Interest Entity [Line Items]
Coupon Rate		7.00%
Maximum | 7.01 – 9.00%
Variable Interest Entity [Line Items]
Coupon Rate	9.00%	9.00%
Maximum | 11.01 – 13.00%
Variable Interest Entity [Line Items]
Coupon Rate	11.00%	11.00%
Maximum | 17.01 - 19.00%
Variable Interest Entity [Line Items]
Coupon Rate	13.00%	13.00%
Maximum | 13.01 - 14.00%
Variable Interest Entity [Line Items]
Coupon Rate	15.00%	15.00%
Maximum | 17.01 – 19.00%
Variable Interest Entity [Line Items]
Coupon Rate	19.00%	19.00%
Maximum | Residential Portfolio Segment Current Coupon Rate Five
Variable Interest Entity [Line Items]
Coupon Rate	8.00%	8.00%
Non-Qualifying Adjustable Rate Mortgage | Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	2,908
Fixed Rate Residential Mortgage Loan Held in Securitization Trusts | Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	612
Investor Fixed Rate Mortgage | Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	9