Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2020 USD ($) $ / loan	Dec. 31, 2019 USD ($)
Liabilities
Contingent consideration | $	$ 0	$ 28,484
ABS Issued Securities
Liabilities
ABS issued | $	$ 1,629,683
ABS Issued Securities | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement inputs	0.08
ABS Issued Securities | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement inputs	0.55
ABS Issued Securities | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement inputs	0.31
ABS Issued Securities | Discount rate | Minimum
Liabilities
ABS issued, measurement inputs	0.02
ABS Issued Securities | Discount rate | Maximum
Liabilities
ABS issued, measurement inputs	0.26
ABS Issued Securities | Discount rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.03
ABS Issued Securities | Default rate | Minimum
Liabilities
ABS issued, measurement inputs	0
ABS Issued Securities | Default rate | Maximum
Liabilities
ABS issued, measurement inputs	0.41
ABS Issued Securities | Default rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.03
ABS Issued Securities | Loss severity | Minimum
Liabilities
ABS issued, measurement inputs	0.30
ABS Issued Securities | Loss severity | Maximum
Liabilities
ABS issued, measurement inputs	0.50
ABS Issued Securities | Loss severity | Weighted Average
Liabilities
ABS issued, measurement inputs	0.31
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,793,620
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement inputs	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement inputs	0.07
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement inputs	0.06
Freddie Mac SLST | Discount rate | Minimum
Liabilities
ABS issued, measurement inputs	0.01
Freddie Mac SLST | Discount rate | Maximum
Liabilities
ABS issued, measurement inputs	0.07
Freddie Mac SLST | Discount rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.02
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement inputs	0.09
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement inputs	0.15
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.11
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement inputs	0.35
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement inputs	0.35
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement inputs	0.35
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 463,966
Freddie Mac K-Series | Discount rate | Minimum
Liabilities
ABS issued, measurement inputs	0.01
Freddie Mac K-Series | Discount rate | Maximum
Liabilities
ABS issued, measurement inputs	0.17
Freddie Mac K-Series | Discount rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.02
CAFL
Liabilities
ABS issued | $	$ 3,013,093
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement inputs	0.03
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement inputs	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement inputs	0.03
CAFL | Discount rate | Minimum
Liabilities
ABS issued, measurement inputs	0.01
CAFL | Discount rate | Maximum
Liabilities
ABS issued, measurement inputs	0.40
CAFL | Discount rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.03
CAFL | Default rate | Minimum
Liabilities
ABS issued, measurement inputs	0
CAFL | Default rate | Maximum
Liabilities
ABS issued, measurement inputs	0.18
CAFL | Default rate | Weighted Average
Liabilities
ABS issued, measurement inputs	0.10
CAFL | Loss severity | Minimum
Liabilities
ABS issued, measurement inputs	0.30
CAFL | Loss severity | Maximum
Liabilities
ABS issued, measurement inputs	0.30
CAFL | Loss severity | Weighted Average
Liabilities
ABS issued, measurement inputs	0.30
Jumbo fixed-rate loans
Assets
Loans, held-for-investment | $	$ 19,464
Jumbo fixed-rate loans | Whole loan spread to swap rate | Minimum
Assets
Loans held-for-sale, measurement input	0.0305
Jumbo fixed-rate loans | Whole loan spread to swap rate | Maximum
Assets
Loans held-for-sale, measurement input	0.0305
Jumbo fixed-rate loans | Whole loan spread to swap rate | Weighted Average
Assets
Loans held-for-sale, measurement input	0.0305
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans held-for-sale, measurement input	0.20
Jumbo loans committed to sell
Assets
Loans, held-for-investment | $	$ 157,140
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Loans held-for-sale, measurement input	102.54
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Loans held-for-sale, measurement input	103.22
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Loans held-for-sale, measurement input	102.83
Legacy Sequoia
Assets
Loans, held-for-investment | $	$ 285,935
Sequoia Choice
Assets
Loans, held-for-investment | $	1,565,322
Liabilities
Fair value of securities owned | $	222,000
Freddie Mac SLST
Assets
Loans, held-for-investment | $	2,221,153
Liabilities
Fair value of securities owned | $	428,000
Single-family rental loans
Assets
Loans, held-for-investment | $	$ 245,394
Single-family rental loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Non-securitizable loan dollar price | Minimum
Assets
Loans receivable, measurement input	86
Single-family rental loans | Non-securitizable loan dollar price | Maximum
Assets
Loans receivable, measurement input	105
Single-family rental loans | Non-securitizable loan dollar price | Weighted Average
Assets
Loans receivable, measurement input	99
Single-family rental loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.25
Single-family rental loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.34
Single-family rental loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.32
Single-family rental loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.09
Single-family rental loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.09
Single-family rental loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.09
CAFL
Assets
Loans, held-for-investment | $	$ 3,249,194
Liabilities
Fair value of securities owned | $	239,000
Residential Bridge, Held-for-investment at fair value
Assets
Loans, held-for-investment | $	$ 641,765
Residential Bridge, Held-for-investment at fair value | Discount rate | Minimum
Assets
Loans receivable, measurement input	0.06
Residential Bridge, Held-for-investment at fair value | Discount rate | Maximum
Assets
Loans receivable, measurement input	0.15
Residential Bridge, Held-for-investment at fair value | Discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.09
Freddie Mac K-Series
Assets
Loans, held-for-investment | $	$ 492,221
Liabilities
Fair value of securities owned | $	$ 28,000
Business purpose loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input	0.0120
Business purpose loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input	0.0190
Business purpose loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0131
Business purpose loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input	0.0160
Business purpose loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input	0.1650
Business purpose loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0389
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 344,125
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities, measurement inputs	0.09
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities, measurement inputs	0.56
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities, measurement inputs	0.28
Trading and AFS securities | Discount rate | Minimum
Assets
Trading and AFS securities, measurement inputs	0.03
Trading and AFS securities | Discount rate | Maximum
Assets
Trading and AFS securities, measurement inputs	0.25
Trading and AFS securities | Discount rate | Weighted Average
Assets
Trading and AFS securities, measurement inputs	0.09
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities, measurement inputs	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities, measurement inputs	0.25
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities, measurement inputs	0.03
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities, measurement inputs	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities, measurement inputs	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities, measurement inputs	0.21
Trading and AFS securities | CRT dollar price | Minimum
Assets
Trading and AFS securities, measurement inputs	84
Trading and AFS securities | CRT dollar price | Maximum
Assets
Trading and AFS securities, measurement inputs	104
Trading and AFS securities | CRT dollar price | Weighted Average
Assets
Trading and AFS securities, measurement inputs	95
Servicer advance investments
Assets
Servicing asset | $	$ 231,489
Servicer advance investments | Minimum
Assets
Expected remaining life	4 years
Servicer advance investments | Maximum
Assets
Expected remaining life	4 years
Servicer advance investments | Weighted Average
Assets
Expected remaining life	4 years
Servicer advance investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing assets, measurement inputs	0.08
Servicer advance investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing assets, measurement inputs	0.14
Servicer advance investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing assets, measurement inputs	0.14
Servicer advance investments | Discount rate | Minimum
Assets
Servicing assets, measurement inputs	0.03
Servicer advance investments | Discount rate | Maximum
Assets
Servicing assets, measurement inputs	0.04
Servicer advance investments | Discount rate | Weighted Average
Assets
Servicing assets, measurement inputs	0.03
MSRs
Assets
Servicing asset | $	$ 8,815
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing assets, measurement inputs	0.10
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing assets, measurement inputs	1
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing assets, measurement inputs	0.38
MSRs | Discount rate | Minimum
Assets
Servicing assets, measurement inputs	0.12
MSRs | Discount rate | Maximum
Assets
Servicing assets, measurement inputs	0.12
MSRs | Discount rate | Weighted Average
Assets
Servicing assets, measurement inputs	0.12
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing assets, measurement inputs	97
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing assets, measurement inputs	97
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing assets, measurement inputs	97
Excess MSRs
Assets
Servicing asset | $	$ 34,418
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing assets, measurement inputs	0.14
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing assets, measurement inputs	0.27
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing assets, measurement inputs	0.19
Excess MSRs | Discount rate | Minimum
Assets
Servicing assets, measurement inputs	0.15
Excess MSRs | Discount rate | Maximum
Assets
Servicing assets, measurement inputs	0.19
Excess MSRs | Discount rate | Weighted Average
Assets
Servicing assets, measurement inputs	0.17
Excess MSRs | Excess mortgage servicing amount | Minimum
Assets
Servicing assets, measurement inputs	0.0009
Excess MSRs | Excess mortgage servicing amount | Maximum
Assets
Servicing assets, measurement inputs	0.0016
Excess MSRs | Excess mortgage servicing amount | Weighted Average
Assets
Servicing assets, measurement inputs	0.0012
Shared home appreciation options
Assets
Shared home appreciation options | $	$ 42,440
Shared home appreciation options | Prepayment rate (annual CPR) | Minimum
Assets
Shared home appreciation options, measurement inputs	0.10
Shared home appreciation options | Prepayment rate (annual CPR) | Maximum
Assets
Shared home appreciation options, measurement inputs	0.24
Shared home appreciation options | Prepayment rate (annual CPR) | Weighted Average
Assets
Shared home appreciation options, measurement inputs	0.18
Shared home appreciation options | Discount rate | Minimum
Assets
Shared home appreciation options, measurement inputs	0.15
Shared home appreciation options | Discount rate | Maximum
Assets
Shared home appreciation options, measurement inputs	0.15
Shared home appreciation options | Discount rate | Weighted Average
Assets
Shared home appreciation options, measurement inputs	0.15
Shared home appreciation options | Home price appreciation | Minimum
Assets
Shared home appreciation options, measurement inputs	0.03
Shared home appreciation options | Home price appreciation | Maximum
Assets
Shared home appreciation options, measurement inputs	0.03
Shared home appreciation options | Home price appreciation | Weighted Average
Assets
Shared home appreciation options, measurement inputs	0.03
REO
Assets
REO | $	$ 1,117
REO | Loss severity | Minimum
Assets
REO, measurement inputs	0.01
REO | Loss severity | Maximum
Assets
REO, measurement inputs	0.46
REO | Loss severity | Weighted Average
Assets
REO, measurement inputs	0.21
Loan purchase and interest rate lock commitments
Assets
Loan purchase commitments, net | $	$ 14,450
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Minimum
Assets
Loan purchase commitments, net, measurement inputs	0.20
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Maximum
Assets
Loan purchase commitments, net, measurement inputs	0.20
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	0.20
Loan purchase and interest rate lock commitments | MSR multiple | Minimum
Assets
Loan purchase commitments, net, measurement inputs	0
Loan purchase and interest rate lock commitments | MSR multiple | Maximum
Assets
Loan purchase commitments, net, measurement inputs	3.9
Loan purchase and interest rate lock commitments | MSR multiple | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	3.0
Loan purchase and interest rate lock commitments | Mortgage servicing income | Minimum
Assets
Loan purchase commitments, net, measurement inputs	0
Loan purchase and interest rate lock commitments | Mortgage servicing income | Maximum
Assets
Loan purchase commitments, net, measurement inputs	0.0016
Loan purchase and interest rate lock commitments | Mortgage servicing income | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	0.0007
Loan purchase and interest rate lock commitments | Pull-through rate | Minimum
Assets
Loan purchase commitments, net, measurement inputs	0.17
Loan purchase and interest rate lock commitments | Pull-through rate | Maximum
Assets
Loan purchase commitments, net, measurement inputs	1
Loan purchase and interest rate lock commitments | Pull-through rate | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	0.65
Loan purchase and interest rate lock commitments | Whole loan spread to TBA price | Minimum
Assets
Loan purchase commitments, net, measurement inputs	2.50
Loan purchase and interest rate lock commitments | Whole loan spread to TBA price | Maximum
Assets
Loan purchase commitments, net, measurement inputs	2.50
Loan purchase and interest rate lock commitments | Whole loan spread to TBA price | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	2.50
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate - fixed rate | Minimum
Assets
Loan purchase commitments, net, measurement inputs	0.0305
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate - fixed rate | Maximum
Assets
Loan purchase commitments, net, measurement inputs	0.0305
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate - fixed rate | Weighted Average
Assets
Loan purchase commitments, net, measurement inputs	0.0305
Loan purchase and interest rate lock commitments | Committed sales price | Minimum
Assets
Loan purchase commitments, measurement input	100.94
Loan purchase and interest rate lock commitments | Committed sales price | Maximum
Assets
Loan purchase commitments, measurement input	103.22
Loan purchase and interest rate lock commitments | Committed sales price | Weighted Average
Assets
Loan purchase commitments, measurement input	102.25