Residential Whole Loans and Bridge Loans - Narrative (Details)	1 Months Ended	3 Months Ended	12 Months Ended
	May 31, 2019 USD ($)	Mar. 31, 2023 USD ($) loan trust property	Dec. 31, 2022 USD ($) trust loan property	Jul. 31, 2022 USD ($)	Feb. 28, 2022 USD ($)	Jun. 30, 2020 USD ($)
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase		$ 171,290,000	$ 193,117,000
Number of real estate properties owned | property		1	1
Residential  Whole Loans
Variable Interest Entity [Line Items]
Number of loans | loan		2,892	2,943
Fair value		$ 1,074,417,000	$ 1,091,145,000
Residential  Whole Loans | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		2,892
Residential Bridge Loans | 90+ days
Variable Interest Entity [Line Items]
Number of loans | loan		4
Residential Bridge Loans | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		4	5
Fair value		$ 2,800,000	$ 2,800,000
Residential Bridge Loans | Investor Fixed Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		5	5
Residential Bridge Loans | Investor Fixed Rate Mortgage Loan | Estimated Fair Value
Variable Interest Entity [Line Items]
Fair value		$ 1,200,000	$ 1,200,000
Residential Bridge Loans
Variable Interest Entity [Line Items]
Fair value		$ 2,782,000
Number of loans | loan		4
Unpaid principal balance		$ 3,091,000
Residential Bridge Loans | 90+ days
Variable Interest Entity [Line Items]
Fair value		$ 2,782,000
Number of loans | loan		4	5
Unpaid principal balance		$ 3,091,000
RMI 2015 Trust | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | trust		6	6
RMI 2015 Trust | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan | Estimated Fair Value
Variable Interest Entity [Line Items]
Fair value		$ 3,400,000	$ 3,200,000
Arroyo Trust 2019 | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		745,000	766,000
Arroyo Trust 2019 | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan | Estimated Fair Value
Variable Interest Entity [Line Items]
Fair value		$ 233,600,000	$ 237,600,000
Arroyo Trust 2019 | Residential  Whole Loans
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase	$ 919,000,000
Arroyo Trust 2019 | Residential  Whole Loans | Arroyo Trust
Variable Interest Entity [Line Items]
Transferred of non-qm residential whole loans	$ 945,500,000
Arroyo Mortgage Trust 2020 | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		424,000	432
Arroyo Mortgage Trust 2020 | Residential  Whole Loans
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase						$ 341,700,000
Arroyo Mortgage Trust 2020 | Residential  Whole Loans | Arroyo Trust
Variable Interest Entity [Line Items]
Line of credit facility, maximum borrowing capacity						$ 355,800,000
Long-term debt, fair value		$ 135,800,000	$ 135,100,000
Arroyo Mortgage Trust 2022 | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		699	705,000
Arroyo Mortgage Trust 2022 | Residential  Whole Loans
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase					$ 398,900,000
Arroyo Mortgage Trust 2022 | Residential  Whole Loans | Arroyo Trust
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase		$ 348,400,000	$ 350,700,000
Line of credit facility, maximum borrowing capacity					$ 432,000,000
Arroyo Mortgage Trust 2022-2 | Arroyo Trust | Non-Qualifying Adjustable Rate Mortgage Loan
Variable Interest Entity [Line Items]
Number of loans | loan		1,013	1,029,000
Arroyo Mortgage Trust 2022-2 | Residential  Whole Loans
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase				$ 351,900,000
Arroyo Mortgage Trust 2022-2 | Residential  Whole Loans | Arroyo Trust
Variable Interest Entity [Line Items]
Securities sold under agreements to repurchase		$ 352,100,000	$ 363,300,000
Line of credit facility, maximum borrowing capacity				$ 402,200,000
VIE | Residential  Whole Loans
Variable Interest Entity [Line Items]
Number of loans | loan		11	13
Mortgage loans on real estate, principal amount of delinquent loans		$ 7,000,000	$ 8,700,000
Mortgage loans on real estate, number of loans, nonperforming, percentage		0.60%	0.80%
Original LTV, collateral dependent		64.50%	60.00%
VIE | Residential  Whole Loans | Estimated Fair Value
Variable Interest Entity [Line Items]
Fair value		$ 6,600,000	$ 8,000,000
VIE | Residential Bridge Loans
Variable Interest Entity [Line Items]
Number of loans | loan		4	5
Unpaid principal balance		$ 3,100,000
Investment in mortgage loans on real estate, principal amount at fair value of delinquent loan			$ 3,200,000
Foreclosed other real estate owned		2,300,000	2,300,000
VIE | Residential Bridge Loans | Estimated Fair Value
Variable Interest Entity [Line Items]
Fair value		$ 2,800,000	$ 2,800,000