Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details)	3 Months Ended
	Mar. 31, 2023 USD ($) $ / loan	Dec. 31, 2022 USD ($)
Assets
ABS issued, net, At fair value | $	$ 7,968,135,000	$ 7,424,132,000
Variable Interest Entity, Primary Beneficiary | Collateralized Financing Entities | HEI
Liabilities
Fair value of securities owned | $	13,000,000
ABS Issued
Liabilities
ABS issued | $	$ 3,739,429,000
ABS Issued | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.04
ABS Issued | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.18
ABS Issued | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.06
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.13
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.01
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.25
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.29
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.05
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.59
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.08
CAFL
Liabilities
ABS issued | $	$ 2,593,192,000
CAFL | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.17
CAFL | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.06
CAFL | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.05
CAFL | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.16
CAFL | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.08
CAFL | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.40
CAFL | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.31
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.001
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,143,522,000
Freddie Mac SLST | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.05
Freddie Mac SLST | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.16
Freddie Mac SLST | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.05
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.13
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.14
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.14
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.07
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 394,469,000
Freddie Mac K-Series | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.03
Freddie Mac K-Series | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.10
Freddie Mac K-Series | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.05
Consolidated Point Entities
Liabilities
ABS issued | $	$ 97,523,000
Consolidated Point Entities | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.14
Consolidated Point Entities | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Home price appreciation (depreciation) | Minimum
Liabilities
ABS issued, measurement input	(0.07)
Consolidated Point Entities | Home price appreciation (depreciation) | Maximum
Liabilities
ABS issued, measurement input	0.04
Consolidated Point Entities | Home price appreciation (depreciation) | Weighted Average
Liabilities
ABS issued, measurement input	0.03
Jumbo fixed-rate loans
Assets
ABS issued, net, At fair value | $	$ 26,975,000
Jumbo fixed-rate loans | Whole loan spread to swap rate | Minimum
Assets
Loans held-for-sale, measurement input	0.0112
Jumbo fixed-rate loans | Whole loan spread to swap rate | Maximum
Assets
Loans held-for-sale, measurement input	0.0112
Jumbo fixed-rate loans | Whole loan spread to swap rate | Weighted Average
Assets
Loans held-for-sale, measurement input	0.0112
Jumbo fixed-rate loans | Seasoned whole loan dollar price | Minimum
Assets
Loans held-for-sale, measurement input	91
Jumbo fixed-rate loans | Seasoned whole loan dollar price | Maximum
Assets
Loans held-for-sale, measurement input	91
Jumbo fixed-rate loans | Seasoned whole loan dollar price | Weighted Average
Assets
Loans held-for-sale, measurement input	91
Legacy Sequoia
Assets
Loans receivable, fair value | $	$ 170,000,000
Sequoia
Assets
Loans receivable, fair value | $	3,831,538,000
Liabilities
Fair value of securities owned | $	261,000,000
Freddie Mac SLST
Assets
Loans receivable, fair value | $	1,464,345,000
Liabilities
Fair value of securities owned | $	323,000,000	323,000,000
BPL term loans
Assets
Loans receivable, fair value | $	$ 354,166,000
BPL term loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input	0.0180
BPL term loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input	0.0180
BPL term loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0180
BPL term loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input	0.0240
BPL term loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input	0.0600
BPL term loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0337
BPL term loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.36
BPL term loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.36
BPL term loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.36
BPL term loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.07
BPL term loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.13
BPL term loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.10
BPL term loans | Whole loan spread to treasury rate | Minimum
Assets
Loans receivable, measurement input	0.0325
BPL term loans | Whole loan spread to treasury rate | Maximum
Assets
Loans receivable, measurement input	0.0550
BPL term loans | Whole loan spread to treasury rate | Weighted Average
Assets
Loans receivable, measurement input	0.0441
BPL term loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.03
BPL term loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.03
BPL term loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.03
CAFL
Assets
Loans receivable, fair value | $	$ 2,891,043,000
Liabilities
Fair value of securities owned | $	295,000,000	$ 304,000,000
BPL bridge loans
Assets
Loans receivable, fair value | $	$ 2,119,440
BPL bridge loans | Senior credit spread | Minimum
Assets
Loans held-for-investment, measurement input	0.0280
BPL bridge loans | Senior credit spread | Maximum
Assets
Loans held-for-investment, measurement input	0.0280
BPL bridge loans | Senior credit spread | Weighted Average
Assets
Loans held-for-investment, measurement input	0.0280
BPL bridge loans | Subordinate credit spread | Minimum
Assets
Loans held-for-investment, measurement input	0.0335
BPL bridge loans | Subordinate credit spread | Maximum
Assets
Loans held-for-investment, measurement input	0.1150
BPL bridge loans | Subordinate credit spread | Weighted Average
Assets
Loans held-for-investment, measurement input	0.0654
BPL bridge loans | Senior credit support | Minimum
Assets
Loans held-for-investment, measurement input	0.43
BPL bridge loans | Senior credit support | Maximum
Assets
Loans held-for-investment, measurement input	0.43
BPL bridge loans | Senior credit support | Weighted Average
Assets
Loans held-for-investment, measurement input	0.43
BPL bridge loans | Whole loan discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.05
BPL bridge loans | Whole loan discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.15
BPL bridge loans | Whole loan discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.10
Freddie Mac K-Series
Assets
Loans receivable, fair value | $	$ 426,599,000
Liabilities
Fair value of securities owned | $	32,000,000
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 243,346,000
Trading and AFS securities | CRT dollar price | Minimum
Assets
Trading and AFS securities	64
Trading and AFS securities | CRT dollar price | Maximum
Assets
Trading and AFS securities	96
Trading and AFS securities | CRT dollar price | Weighted Average
Assets
Trading and AFS securities	87
Trading and AFS securities | Whole loan discount rate | Minimum
Assets
Trading and AFS securities	0.05
Trading and AFS securities | Whole loan discount rate | Maximum
Assets
Trading and AFS securities	0.18
Trading and AFS securities | Whole loan discount rate | Weighted Average
Assets
Trading and AFS securities	0.10
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.12
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.005
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.25
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0.05
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.65
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.09
HEIs
Assets
HEIs | $	$ 287,466,000
HEIs | Whole loan discount rate | Minimum
Assets
HEIs, Measurement input	0.10
HEIs | Whole loan discount rate | Maximum
Assets
HEIs, Measurement input	0.10
HEIs | Whole loan discount rate | Weighted Average
Assets
HEIs, Measurement input	0.10
HEIs | Prepayment rate (annual CPR) | Minimum
Assets
HEIs, Measurement input	0.01
HEIs | Prepayment rate (annual CPR) | Maximum
Assets
HEIs, Measurement input	0.23
HEIs | Prepayment rate (annual CPR) | Weighted Average
Assets
HEIs, Measurement input	0.16
HEIs | Home price appreciation (depreciation) | Minimum
Assets
HEIs, Measurement input	(0.07)
HEIs | Home price appreciation (depreciation) | Maximum
Assets
HEIs, Measurement input	0.04
HEIs | Home price appreciation (depreciation) | Weighted Average
Assets
HEIs, Measurement input	0.03
HEIs held by Point HEI entity
Assets
HEIs | $	$ 129,317,000
Servicer Advance Investments
Assets
Servicing asset | $	$ 260,378,000
Servicer Advance Investments | Minimum
Assets
Expected remaining life	5 years
Servicer Advance Investments | Maximum
Assets
Expected remaining life	6 years
Servicer Advance Investments | Weighted Average
Assets
Expected remaining life	5 years
Servicer Advance Investments | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.02
Servicer Advance Investments | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.04
Servicer Advance Investments | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.03
Servicer Advance Investments | Mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0
Servicer Advance Investments | Mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0018
Servicer Advance Investments | Mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0003
Servicer Advance Investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.14
Servicer Advance Investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.30
Servicer Advance Investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.14
MSRs
Assets
Servicing asset | $	$ 24,831,000
MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.11
MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.53
MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.11
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing asset, measurement input	93
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing asset, measurement input	93
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing asset, measurement input	93
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.04
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.27
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.08
Excess MSRs
Assets
Servicing asset | $	$ 38,807,000
Excess MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.13
Excess MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.19
Excess MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Excess mortgage servicing amount | Minimum
Assets
Servicing asset, measurement input	0.0008
Excess MSRs | Excess mortgage servicing amount | Maximum
Assets
Servicing asset, measurement input	0.0019
Excess MSRs | Excess mortgage servicing amount | Weighted Average
Assets
Servicing asset, measurement input	0.0011
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.10
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	1
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.17
Residential loan purchase commitments
Liabilities
Loan purchase commitments, net | $	$ 359,000
Residential loan purchase commitments | Whole loan spread to swap rate | Minimum
Assets
Loans receivable, measurement input	0.0112
Residential loan purchase commitments | Whole loan spread to swap rate | Maximum
Assets
Loans receivable, measurement input	0.0137
Residential loan purchase commitments | Whole loan spread to swap rate | Weighted Average
Assets
Loans receivable, measurement input	0.0125
Residential loan purchase commitments | Pull-through rate | Minimum
Assets
Loans receivable, measurement input	0.36
Residential loan purchase commitments | Pull-through rate | Maximum
Assets
Loans receivable, measurement input	1
Residential loan purchase commitments | Pull-through rate | Weighted Average
Assets
Loans receivable, measurement input	0.69
Residential loan purchase commitments | Committed sales price | Minimum
Assets
Loans receivable, measurement input	99
Residential loan purchase commitments | Committed sales price | Maximum
Assets
Loans receivable, measurement input	103
Residential loan purchase commitments | Committed sales price | Weighted Average
Assets
Loans receivable, measurement input	101