Schedule IV - Mortgage Loans On Real Estate (Details)	12 Months Ended
	Dec. 31, 2022 USD ($) loan	Dec. 31, 2022 USD ($)	Dec. 31, 2022 USD ($) tranche	Dec. 31, 2021 USD ($) loan tranche	Dec. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	$ 11,370,323,000	$ 11,370,323,000	$ 11,370,323,000	$ 12,856,934,000	$ 8,877,626,000	$ 15,630,117,000
Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	4,832,407,000	4,832,407,000	4,832,407,000
Principal Amount Subject to Delinquent Principal or Interest	224,020,000	224,020,000	$ 224,020,000
Residential loans, held-for-sale, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | tranche			994	2,196
Carrying Amount	780,781,000	780,781,000	$ 780,781,000
Principal Amount Subject to Delinquent Principal or Interest	$ 208,000	208,000	$ 208,000	$ 2,923,000
Residential loans, held-for-sale, at fair value | Hybrid ARM loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans	8		8	57
Carrying Amount	$ 4,130,000	4,130,000	$ 4,130,000
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	$ 0	$ 0
Residential loans, held-for-sale, at fair value | Hybrid ARM loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.63%
Residential loans, held-for-sale, at fair value | Hybrid ARM loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.50%
Residential loans, held-for-sale, at fair value | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans	986		986	2,139
Carrying Amount	$ 776,651,000	$ 776,651,000	$ 776,651,000
Principal Amount Subject to Delinquent Principal or Interest	208,000	$ 208,000	208,000	$ 2,923,000
Residential loans, held-for-sale, at fair value | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		2.75%
Residential loans, held-for-sale, at fair value | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		9.25%
Single-family rental loans, held-for-sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	358,791,000	$ 358,791,000	358,791,000
Principal Amount Subject to Delinquent Principal or Interest	$ 536,000	536,000	536,000
Single-family rental loans, held-for-sale | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	91
Carrying Amount	$ 358,791,000	358,791,000	358,791,000
Principal Amount Subject to Delinquent Principal or Interest	536,000	$ 536,000	536,000
Single-family rental loans, held-for-sale | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.75%
Single-family rental loans, held-for-sale | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		8.47%
Single family rental loans held for investment at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	2,944,984,000	$ 2,944,984,000	2,944,984,000
Principal Amount Subject to Delinquent Principal or Interest	39,483,000	39,483,000	39,483,000
Business purpose loans held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	1,512,427,000	1,512,427,000	1,512,427,000
Principal Amount Subject to Delinquent Principal or Interest	34,264,000	34,264,000	34,264,000
Real estate loans, federal income tax basis	$ 2,030,000,000.00	2,030,000,000.00	2,030,000,000.00
Business purpose loans held-for-investment | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	261
Carrying Amount	$ 99,974,000	99,974,000	99,974,000
Principal Amount Subject to Delinquent Principal or Interest	$ 33,392,000	$ 33,392,000	33,392,000
Business purpose loans held-for-investment | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.00%
Business purpose loans held-for-investment | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		11.50%
Business purpose loans held-for-investment | Floating ARM Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	1,340
Carrying Amount	$ 1,412,453,000	$ 1,412,453,000	1,412,453,000
Principal Amount Subject to Delinquent Principal or Interest	872,000	$ 872,000	872,000
Business purpose loans held-for-investment | Floating ARM Loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		8.27%
Business purpose loans held-for-investment | Floating ARM Loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		11.87%
Consolidated Agency multifamily loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	424,552,000	$ 424,552,000	424,552,000
Principal Amount Subject to Delinquent Principal or Interest	$ 0	0	0
Consolidated Agency multifamily loans | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	28
Carrying Amount	$ 424,552,000	424,552,000	424,552,000
Principal Amount Subject to Delinquent Principal or Interest	0	$ 0	0
Consolidated Agency multifamily loans | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		4.25%
Consolidated Agency multifamily loans | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		4.25%
Legacy Sequoia
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Real estate loans, federal income tax basis	$ 0	$ 0	0
Legacy Sequoia | Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	1,304			1,583
Principal Amount Subject to Delinquent Principal or Interest	$ 6,824,000	6,824,000	6,824,000	$ 7,482,000
Legacy Sequoia | Residential loans, held-for-investment | ARM loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	1,297			1,574
Carrying Amount	$ 183,204,000	183,204,000	183,204,000
Principal Amount Subject to Delinquent Principal or Interest	$ 6,824,000	$ 6,824,000	6,824,000	$ 7,482,000
Legacy Sequoia | Residential loans, held-for-investment | ARM loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		1.25%
Legacy Sequoia | Residential loans, held-for-investment | ARM loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.13%
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	7			9
Carrying Amount	$ 1,729,000	$ 1,729,000	1,729,000
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0	$ 0
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		2.88%
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		4.63%
Sequoia | Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	4,624			4,300
Principal Amount Subject to Delinquent Principal or Interest	$ 7,799,000	$ 7,799,000	7,799,000	$ 15,124,000
Sequoia | Residential loans, held-for-investment | Hybrid ARM loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	17			28
Carrying Amount	$ 10,959,000	10,959,000	10,959,000
Principal Amount Subject to Delinquent Principal or Interest	$ 637,000	$ 637,000	637,000	$ 0
Sequoia | Residential loans, held-for-investment | Hybrid ARM loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.38%
Sequoia | Residential loans, held-for-investment | Hybrid ARM loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		5.63%
Sequoia | Residential loans, held-for-investment | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	4,607			4,272
Carrying Amount	$ 3,179,457,000	$ 3,179,457,000	3,179,457,000
Principal Amount Subject to Delinquent Principal or Interest	7,162,000	$ 7,162,000	7,162,000	$ 15,124,000
Sequoia | Residential loans, held-for-investment | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		1.88%
Sequoia | Residential loans, held-for-investment | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.75%
Freddie Mac SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Real estate loans, federal income tax basis	$ 0	$ 0	0
Freddie Mac SLST | Residential loans, held-for-investment | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	10,882			11,986
Carrying Amount	$ 1,457,058,000	1,457,058,000	1,457,058,000
Principal Amount Subject to Delinquent Principal or Interest	$ 209,397,000	$ 209,397,000	209,397,000	$ 212,961,000
Freddie Mac SLST | Residential loans, held-for-investment | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		2.00%
Freddie Mac SLST | Residential loans, held-for-investment | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		11.00%
CAFL | Single family rental loans held for investment at fair value | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	1,131
Carrying Amount	$ 2,944,984,000	$ 2,944,984,000	2,944,984,000
Principal Amount Subject to Delinquent Principal or Interest	39,483,000	$ 39,483,000	39,483,000
CAFL | Single family rental loans held for investment at fair value | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.81%
CAFL | Single family rental loans held for investment at fair value | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		7.57%
CAFL | Business purpose loans held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Amount	516,383,000	$ 516,383,000	516,383,000
Principal Amount Subject to Delinquent Principal or Interest	$ 7,328,000	7,328,000	7,328,000
CAFL | Business purpose loans held-for-investment | Fixed loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	605
Carrying Amount	$ 110,869,000	110,869,000	110,869,000
Principal Amount Subject to Delinquent Principal or Interest	$ 3,953,000	$ 3,953,000	3,953,000
CAFL | Business purpose loans held-for-investment | Fixed loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.30%
CAFL | Business purpose loans held-for-investment | Fixed loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		11.24%
CAFL | Business purpose loans held-for-investment | Floating ARM Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan	1,270
Carrying Amount	$ 405,514,000	$ 405,514,000	405,514,000
Principal Amount Subject to Delinquent Principal or Interest	$ 3,375,000	$ 3,375,000	$ 3,375,000
CAFL | Business purpose loans held-for-investment | Floating ARM Loans | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		6.12%
CAFL | Business purpose loans held-for-investment | Floating ARM Loans | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate		12.62%