Residential Whole Loans and Bridge Loans - Collateral Securing (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2023	Dec. 31, 2022
Residential  Whole Loans
Variable Interest Entity [Line Items]
Principal  Balance	$ 1,134,794	$ 1,165,301
Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	100.00%	100.00%
Principal  Balance	$ 1,134,794	$ 1,165,301
Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
State Concentration	100.00%	100.00%
Principal  Balance	$ 3,091	$ 3,166
California | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	67.00%	66.80%
Principal  Balance	$ 760,577	$ 778,732
California | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
State Concentration	56.70%	55.40%
Principal  Balance	$ 1,754	$ 1,754
New York | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	9.20%	9.30%
Principal  Balance	$ 103,907	$ 108,108
New York | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
State Concentration	43.30%	42.20%
Principal  Balance	$ 1,337	$ 1,337
Texas | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	4.80%	4.80%
Principal  Balance	$ 54,646	$ 56,126
Florida | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	4.10%	4.10%
Principal  Balance	$ 46,332	$ 47,681
Georgia | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	3.50%	3.50%
Principal  Balance	$ 39,725	$ 40,845
Other | Geographic Concentration Risk | Financing Receivables Total | Residential  Whole Loans
Variable Interest Entity [Line Items]
State Concentration	11.40%	11.50%
Principal  Balance	$ 129,607	$ 133,809
New Jersey | Geographic Concentration Risk | Financing Receivables Total | Residential Bridge Loans
Variable Interest Entity [Line Items]
State Concentration		2.40%
Principal  Balance		$ 75