Residential Whole Loans and Bridge Loans - Investment Portfolio (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2021 USD ($) fICO_Score loan	Dec. 31, 2020 USD ($) fICO_Score director loan
Variable Interest Entity [Line Items]
Principal balance	$ 2,572,727
Residential Whole Loans
Variable Interest Entity [Line Items]
Number of Loans	2,361	2,497
Principal balance	$ 989,143	$ 984,555
Original LTV	64.80%	62.90%
Original FICO Score | fICO_Score	746	744
Expected Life (in years)	3 years 1 month 6 days	2 years 10 months 24 days
Contractual Maturity (years)	27 years 8 months 12 days	27 years 6 months
Coupon Rate	4.60%	5.10%
2.01% - 3.00%
Variable Interest Entity [Line Items]
Number of Loans	27	4
Principal balance	$ 15,640	$ 3,239
Original LTV	65.10%	66.70%
Original FICO Score | fICO_Score	757	733
Expected Life (in years)	5 years 3 months 18 days	5 years 10 months 24 days
Contractual Maturity (years)	28 years 9 months 18 days	28 years
Coupon Rate	2.80%	2.70%
2.01% - 3.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	2.01%	2.01%
2.01% - 3.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	3.00%	3.00%
3.01% - 4.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	496	118
Principal balance	$ 244,022	$ 41,489
Original LTV	63.70%	55.80%
Original FICO Score | fICO_Score	756	709
Expected Life (in years)	3 years 3 months 18 days	3 years 9 months 18 days
Contractual Maturity (years)	28 years	23 years 3 months 18 days
Coupon Rate	3.70%	3.70%
3.01% - 4.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	3.01%	3.01%
3.01% - 4.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	4.00%	4.00%
4.01% - 5.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	1,051	1,172
Principal balance	$ 413,451	$ 403,398
Original LTV	65.10%	61.80%
Original FICO Score | fICO_Score	747	751
Expected Life (in years)	2 years 10 months 24 days	2 years 8 months 12 days
Contractual Maturity (years)	28 years 2 months 12 days	27 years 8 months 12 days
Coupon Rate	4.70%	4.90%
4.01% - 5.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	4.01%	4.01%
4.01% - 5.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	5.00%	5.00%
5.01% - 6.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	757	1,166
Principal balance	$ 305,344	$ 523,105
Original LTV	64.90%	64.20%
Original FICO Score | fICO_Score	738	740
Expected Life (in years)	3 years	2 years 10 months 24 days
Contractual Maturity (years)	26 years 9 months 18 days	27 years 8 months 12 days
Coupon Rate	5.40%	5.40%
5.01% - 6.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	5.01%	5.01%
5.01% - 6.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	6.00%	6.00%
6.01% - 7.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	28	35
Principal balance	$ 10,181	$ 12,813
Original LTV	67.90%	67.50%
Original FICO Score | fICO_Score	721	720
Expected Life (in years)	3 years 1 month 6 days	3 years 2 months 12 days
Contractual Maturity (years)	25 years 9 months 18 days	27 years
Coupon Rate	6.30%	6.30%
6.01% - 7.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	6.01%	6.01%
6.01% - 7.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	7.00%	7.00%
7.01% - 8.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	2	2
Principal balance	$ 505	$ 511
Original LTV	73.20%	73.20%
Original FICO Score | fICO_Score	753	753
Expected Life (in years)	4 years 6 months	4 years 1 month 6 days
Contractual Maturity (years)	26 years 9 months 18 days	27 years 7 months 6 days
Coupon Rate	7.10%	7.10%
7.01% - 8.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	7.01%	7.01%
7.01% - 8.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	8.00%	8.00%
Residential portfolio segment with no FICO score
Variable Interest Entity [Line Items]
Number of Loans | loan	230	236
Principal balance	$ 74,300	$ 75,200
Residential Bridge Loans | VIE
Variable Interest Entity [Line Items]
Number of Loans | loan	9	29
Principal balance	$ 5,834	$ 15,247
Original LTV	72.90%	72.00%
Contractual Maturity (years)	0 years	9 months 18 days
Coupon Rate	9.70%	9.40%
7.01% – 9.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	3	10
Principal balance	$ 2,946	$ 8,295
Original LTV	70.40%	69.60%
Contractual Maturity (years)	0 years	1 year 4 months 24 days
Coupon Rate	8.80%	8.70%
7.01% – 9.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	7.01%	7.01%
7.01% – 9.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	9.00%	9.00%
9.01% – 11.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	4	15
Principal balance	$ 2,393	$ 6,123
Original LTV	76.70%	75.50%
Contractual Maturity (years)	0 years	6 months
Coupon Rate	10.40%	10.10%
9.01% – 11.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	9.01%	9.01%
9.01% – 11.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	11.00%	11.00%
11.01% - 13.00%
Variable Interest Entity [Line Items]
Number of Loans | loan	2	3
Principal balance	$ 495	$ 705
Original LTV	69.70%	69.80%
Contractual Maturity (years)	0 years	0 years
Coupon Rate	11.40%	11.40%
11.01% - 13.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate	11.01%	11.01%
11.01% - 13.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate	13.00%	13.00%
17.01% – 19.00%
Variable Interest Entity [Line Items]
Number of Loans | loan		1
Principal balance		$ 124
Original LTV		75.00%
Contractual Maturity (years)		0 years
Coupon Rate		18.00%
17.01% – 19.00% | Minimum
Variable Interest Entity [Line Items]
Coupon Rate		17.01%
17.01% – 19.00% | Maximum
Variable Interest Entity [Line Items]
Coupon Rate		19.00%