Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2022 USD ($) $ / loan	Dec. 31, 2021 USD ($)
Assets
ABS issued, net, Fair value | $	$ 8,475,868	$ 8,843,147
Variable Interest Entity, Primary Beneficiary | Collateralized Financing Entities | HEI
Liabilities
Fair value of securities owned | $	14,000
ABS Issued
Liabilities
ABS issued | $	$ 3,848,221
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.07
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.45
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.25
ABS Issued | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.18
ABS Issued | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.05
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.11
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.01
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.25
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.32
CAFL
Liabilities
ABS issued | $	$ 2,745,104
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
CAFL | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.12
CAFL | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.05
CAFL | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.05
CAFL | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.20
CAFL | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.06
CAFL | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,341,282
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.07
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.03
Freddie Mac SLST | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.07
Freddie Mac SLST | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.04
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.10
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.35
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 419,883
Freddie Mac K-Series | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.03
Freddie Mac K-Series | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.08
Freddie Mac K-Series | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.03
Consolidated Point Entities
Liabilities
ABS issued | $	$ 121,378
Consolidated Point Entities | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.07
Consolidated Point Entities | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.12
Consolidated Point Entities | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.07
Consolidated Point Entities | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
Consolidated Point Entities | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.30
Consolidated Point Entities | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
Consolidated Point Entities | Home price appreciation | Minimum
Liabilities
ABS issued, measurement input	0.03
Consolidated Point Entities | Home price appreciation | Maximum
Liabilities
ABS issued, measurement input	0.04
Consolidated Point Entities | Home price appreciation | Weighted Average
Liabilities
ABS issued, measurement input	0.03
Jumbo fixed-rate loans
Assets
ABS issued, net, Fair value | $	$ 772,917
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Whole Loan Spread to TBA Price | Minimum
Assets
Loans held-for-sale, measurement input	2.94
Jumbo fixed-rate loans | Whole Loan Spread to TBA Price | Maximum
Assets
Loans held-for-sale, measurement input	2.94
Jumbo fixed-rate loans | Whole Loan Spread to TBA Price | Weighted Average
Assets
Loans held-for-sale, measurement input	2.94
Jumbo fixed-rate loans | Whole loan spread to swap rate | Minimum
Assets
Loans held-for-sale, measurement input	0.0225
Jumbo fixed-rate loans | Whole loan spread to swap rate | Maximum
Assets
Loans held-for-sale, measurement input	0.0225
Jumbo fixed-rate loans | Whole loan spread to swap rate | Weighted Average
Assets
Loans held-for-sale, measurement input	0.0225
Jumbo fixed-rate loans | CRT dollar price | Minimum
Assets
Loans held-for-sale, measurement input	100
Jumbo fixed-rate loans | CRT dollar price | Maximum
Assets
Loans held-for-sale, measurement input	100
Jumbo fixed-rate loans | CRT dollar price | Weighted Average
Assets
Loans held-for-sale, measurement input	100
Jumbo loans committed to sell
Assets
Loans receivable, fair value | $	$ 574,119
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Loans held-for-sale, measurement input	95.92
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Loans held-for-sale, measurement input	102.19
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Loans held-for-sale, measurement input	98.23
Legacy Sequoia
Assets
Loans receivable, fair value | $	$ 222,213
Sequoia
Assets
Loans receivable, fair value | $	3,872,316
Liabilities
Fair value of securities owned | $	246,000
Freddie Mac SLST
Assets
Loans receivable, fair value | $	1,775,539
Liabilities
Fair value of securities owned | $	437,000	445,000
Single-family rental loans
Assets
Loans receivable, fair value | $	$ 505,889
Single-family rental loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.25
Single-family rental loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.07
Single-family rental loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input	0.0140
Single-family rental loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input	0.0175
Single-family rental loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0155
Single-family rental loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input	0.0180
Single-family rental loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input	0.1434
Single-family rental loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0377
Single-family rental loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.36
Single-family rental loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.37
Single-family rental loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.36
Single-family rental loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.08
Single-family rental loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.14
Single-family rental loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.09
Single-family rental loans | Non-securitizable loan dollar price | Minimum
Assets
Loans receivable, measurement input	78
Single-family rental loans | Non-securitizable loan dollar price | Maximum
Assets
Loans receivable, measurement input	100
Single-family rental loans | Non-securitizable loan dollar price | Weighted Average
Assets
Loans receivable, measurement input	98
CAFL
Assets
Loans receivable, fair value | $	$ 3,038,939
Liabilities
Fair value of securities owned | $	306,000	$ 302,000
Bridge loans
Assets
Loans receivable, fair value | $	$ 1,209,950
Bridge loans | Senior credit spread | Minimum
Assets
Loans held-for-investment, measurement input	0.0180
Bridge loans | Senior credit spread | Maximum
Assets
Loans held-for-investment, measurement input	0.0180
Bridge loans | Senior credit spread | Weighted Average
Assets
Loans held-for-investment, measurement input	0.0180
Bridge loans | Subordinate credit spread | Minimum
Assets
Loans held-for-investment, measurement input	0.0225
Bridge loans | Subordinate credit spread | Maximum
Assets
Loans held-for-investment, measurement input	0.0790
Bridge loans | Subordinate credit spread | Weighted Average
Assets
Loans held-for-investment, measurement input	0.0498
Bridge loans | Senior credit support | Minimum
Assets
Loans held-for-investment, measurement input	0.47
Bridge loans | Senior credit support | Maximum
Assets
Loans held-for-investment, measurement input	0.47
Bridge loans | Senior credit support | Weighted Average
Assets
Loans held-for-investment, measurement input	0.47
Bridge loans | IO discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.11
Bridge loans | IO discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.11
Bridge loans | IO discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.11
Bridge loans | Whole loan discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.04
Bridge loans | Whole loan discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.15
Bridge loans | Whole loan discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.07
Freddie Mac K-Series
Assets
Loans receivable, fair value | $	$ 451,804
Liabilities
Fair value of securities owned | $	32,000
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 358,882
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0.08
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.55
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.20
Trading and AFS securities | CRT dollar price | Minimum
Assets
Trading and AFS securities	88
Trading and AFS securities | CRT dollar price | Maximum
Assets
Trading and AFS securities	106
Trading and AFS securities | CRT dollar price | Weighted Average
Assets
Trading and AFS securities	95
Trading and AFS securities | Whole loan discount rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Whole loan discount rate | Maximum
Assets
Trading and AFS securities	0.18
Trading and AFS securities | Whole loan discount rate | Weighted Average
Assets
Trading and AFS securities	0.08
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.27
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.04
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.25
Servicer Advance Investments
Assets
Servicing asset | $	$ 302,837
Servicer Advance Investments | Minimum
Assets
Expected remaining life	5 years
Servicer Advance Investments | Maximum
Assets
Expected remaining life	5 years
Servicer Advance Investments | Weighted Average
Assets
Expected remaining life	5 years
Servicer Advance Investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.14
Servicer Advance Investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.30
Servicer Advance Investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.17
Servicer Advance Investments | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.02
Servicer Advance Investments | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.04
Servicer Advance Investments | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.03
Servicer Advance Investments | Mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0
Servicer Advance Investments | Mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0017
Servicer Advance Investments | Mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0005
MSRs
Assets
Servicing asset | $	$ 19,739
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.10
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.29
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.16
MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.12
MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.12
MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.12
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing asset, measurement input	94
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing asset, measurement input	94
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing asset, measurement input	94
Excess MSRs
Assets
Servicing asset | $	$ 43,023
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.31
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.21
Excess MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.13
Excess MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.19
Excess MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Excess mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0.0010
Excess MSRs | Excess mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0017
Excess MSRs | Excess mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0011
Home Appreciation Option
Assets
HEIs | $	$ 73,886
Home Appreciation Option | Prepayment rate (annual CPR) | Minimum
Assets
HEIs, Measurement input	0.01
Home Appreciation Option | Prepayment rate (annual CPR) | Maximum
Assets
HEIs, Measurement input	0.24
Home Appreciation Option | Prepayment rate (annual CPR) | Weighted Average
Assets
HEIs, Measurement input	0.17
Home Appreciation Option | Whole loan discount rate | Minimum
Assets
HEIs, Measurement input	0.10
Home Appreciation Option | Whole loan discount rate | Maximum
Assets
HEIs, Measurement input	0.10
Home Appreciation Option | Whole loan discount rate | Weighted Average
Assets
HEIs, Measurement input	0.10
Home Appreciation Option | Home price appreciation | Minimum
Assets
HEIs, Measurement input	0.03
Home Appreciation Option | Home price appreciation | Maximum
Assets
HEIs, Measurement input	0.04
Home Appreciation Option | Home price appreciation | Weighted Average
Assets
HEIs, Measurement input	0.03
HEIs held by Point HEI entity
Assets
HEIs | $	$ 153,247
REO
Assets
REO | $	$ 646
REO | Loss severity | Minimum
Assets
REO, measurement input	0.04
REO | Loss severity | Maximum
Assets
REO, measurement input	0.24
REO | Loss severity | Weighted Average
Assets
REO, measurement input	0.12
Residential loan purchase commitments, net
Assets
Loan purchase commitments, net | $	$ 12,322
Residential loan purchase commitments, net | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net | Whole Loan Spread to TBA Price | Minimum
Assets
Loans receivable, measurement input	2.94
Residential loan purchase commitments, net | Whole Loan Spread to TBA Price | Maximum
Assets
Loans receivable, measurement input	2.94
Residential loan purchase commitments, net | Whole Loan Spread to TBA Price | Weighted Average
Assets
Loans receivable, measurement input	2.94
Residential loan purchase commitments, net | Whole loan spread to swap rate | Minimum
Assets
Loans receivable, measurement input	0.0225
Residential loan purchase commitments, net | Whole loan spread to swap rate | Maximum
Assets
Loans receivable, measurement input	0.0225
Residential loan purchase commitments, net | Whole loan spread to swap rate | Weighted Average
Assets
Loans receivable, measurement input	0.0225
Residential loan purchase commitments, net | Pull-through rate | Minimum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net | Pull-through rate | Maximum
Assets
Loans receivable, measurement input	1
Residential loan purchase commitments, net | Pull-through rate | Weighted Average
Assets
Loans receivable, measurement input	0.73
Residential loan purchase commitments, net | Committed sales price | Minimum
Assets
Loans receivable, measurement input	95.92
Residential loan purchase commitments, net | Committed sales price | Maximum
Assets
Loans receivable, measurement input	102.29
Residential loan purchase commitments, net | Committed sales price | Weighted Average
Assets
Loans receivable, measurement input	98.18
Single Family Rental Interest Rate Lock Commitments
Assets
Derivative asset, fair value | $	$ 59
Single Family Rental Interest Rate Lock Commitments | Prepayment rate (annual CPR) | Minimum
Assets
Derivative asset, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments | Prepayment rate (annual CPR) | Maximum
Assets
Derivative asset, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments | Prepayment rate (annual CPR) | Weighted Average
Assets
Derivative asset, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments | Senior credit spread | Minimum
Assets
Derivative asset, measurement input	0.0140
Single Family Rental Interest Rate Lock Commitments | Senior credit spread | Maximum
Assets
Derivative asset, measurement input	0.0140
Single Family Rental Interest Rate Lock Commitments | Senior credit spread | Weighted Average
Assets
Derivative asset, measurement input	0.0140
Single Family Rental Interest Rate Lock Commitments | Subordinate credit spread | Minimum
Assets
Derivative asset, measurement input	0.0180
Single Family Rental Interest Rate Lock Commitments | Subordinate credit spread | Maximum
Assets
Derivative asset, measurement input	0.1434
Single Family Rental Interest Rate Lock Commitments | Subordinate credit spread | Weighted Average
Assets
Derivative asset, measurement input	0.0331
Single Family Rental Interest Rate Lock Commitments | Senior credit support | Minimum
Assets
Derivative asset, measurement input	0.37
Single Family Rental Interest Rate Lock Commitments | Senior credit support | Maximum
Assets
Derivative asset, measurement input	0.37
Single Family Rental Interest Rate Lock Commitments | Senior credit support | Weighted Average
Assets
Derivative asset, measurement input	0.37
Single Family Rental Interest Rate Lock Commitments | IO discount rate | Minimum
Assets
Derivative asset, measurement input	0.08
Single Family Rental Interest Rate Lock Commitments | IO discount rate | Maximum
Assets
Derivative asset, measurement input	0.08
Single Family Rental Interest Rate Lock Commitments | IO discount rate | Weighted Average
Assets
Derivative asset, measurement input	0.08
Single Family Rental Interest Rate Lock Commitments | Pull-through rate | Minimum
Assets
Derivative asset, measurement input	1
Single Family Rental Interest Rate Lock Commitments | Pull-through rate | Maximum
Assets
Derivative asset, measurement input	1
Single Family Rental Interest Rate Lock Commitments | Pull-through rate | Weighted Average
Assets
Derivative asset, measurement input	1