Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	9 Months Ended
	Sep. 30, 2020 USD ($) $ / loan	Dec. 31, 2019 USD ($)	Sep. 30, 2019
Assets
ABS issued, net, Fair value | $	$ 6,969,376	$ 10,515,475
ABS Issued
Liabilities
ABS issued | $	$ 1,919,048
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.05
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.53
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.27
ABS Issued | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
ABS Issued | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.30
ABS Issued | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.03
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.40
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.02
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.31
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,841,313
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 464,865
Freddie Mac K-Series | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.18
Freddie Mac K-Series | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.02
Freddie Mac K-Series | Dollar price | Minimum
Liabilities
ABS issued, measurement input	1
Freddie Mac K-Series | Dollar price | Maximum
Liabilities
ABS issued, measurement input	108
Freddie Mac K-Series | Dollar price | Weighted Average
Liabilities
ABS issued, measurement input	99
CAFL
Liabilities
ABS issued | $	$ 2,744,150
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0
CAFL | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.002
CAFL | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.40
CAFL | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.18
CAFL | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.11
CAFL | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
Jumbo fixed-rate loans
Assets
ABS issued, net, Fair value | $	$ 6,312
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans | Whole loan spread to swap rate | Minimum
Assets
Loans held-for-sale, measurement input	0.0350
Jumbo fixed-rate loans | Whole loan spread to swap rate | Maximum
Assets
Loans held-for-sale, measurement input	0.0350
Jumbo fixed-rate loans | Whole loan spread to swap rate | Weighted Average
Assets
Loans held-for-sale, measurement input	0.0350
Jumbo loans committed to sell
Assets
Loans receivable, fair value | $	$ 98,779
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Loans held-for-sale, measurement input	101.61
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Loans held-for-sale, measurement input	103.40
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Loans held-for-sale, measurement input	103.23
Legacy Sequoia
Assets
Loans receivable, fair value | $	$ 296,765
Sequoia Choice
Assets
Loans receivable, fair value | $	1,836,361
Liabilities
Fair value of securities owned | $	215,000
Freddie Mac SLST
Assets
Loans receivable, fair value | $	2,256,682
Liabilities
Fair value of securities owned | $	416,000
Single-family rental loans
Assets
Loans receivable, fair value | $	$ 285,549
Single-family rental loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0
Single-family rental loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input	0.0130
Single-family rental loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input	0.0130
Single-family rental loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0130
Single-family rental loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input	0.0200
Single-family rental loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input	0.1600
Single-family rental loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0551
Single-family rental loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.30
Single-family rental loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.32
Single-family rental loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.31
Single-family rental loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.08
Single-family rental loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.09
Single-family rental loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.09
Single-family rental loans | Non-securitizable loan dollar price | Minimum
Assets
Loans receivable, measurement input	101
Single-family rental loans | Non-securitizable loan dollar price | Maximum
Assets
Loans receivable, measurement input	101
Single-family rental loans | Non-securitizable loan dollar price | Weighted Average
Assets
Loans receivable, measurement input	101
CAFL
Assets
Loans receivable, fair value | $	$ 2,969,692
Liabilities
Fair value of securities owned | $	229,000
Residential bridge loans
Assets
Loans receivable, fair value | $	$ 700,860
Residential bridge loans | Discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.06
Residential bridge loans | Discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.12
Residential bridge loans | Discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.08
Residential bridge loans | Non-performing loan dollar price | Minimum
Assets
Loans held-for-investment, measurement input	3
Residential bridge loans | Non-performing loan dollar price | Maximum
Assets
Loans held-for-investment, measurement input	100
Residential bridge loans | Non-performing loan dollar price | Weighted Average
Assets
Loans held-for-investment, measurement input	89
Freddie Mac K-Series
Assets
Loans receivable, fair value | $	$ 491,415
Liabilities
Fair value of securities owned | $	27,000
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 351,335
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0.07
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.65
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.24
Trading and AFS securities | Discount rate | Minimum
Assets
Trading and AFS securities	0.03
Trading and AFS securities | Discount rate | Maximum
Assets
Trading and AFS securities	0.34
Trading and AFS securities | Discount rate | Weighted Average
Assets
Trading and AFS securities	0.09
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.26
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.01
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.19
Trading and AFS securities | Measurement Input, CRT Dollar Price | Minimum
Assets
Trading and AFS securities	49
Trading and AFS securities | Measurement Input, CRT Dollar Price | Maximum
Assets
Trading and AFS securities	103
Trading and AFS securities | Measurement Input, CRT Dollar Price | Weighted Average
Assets
Trading and AFS securities	84
Servicer advance investments
Assets
Servicing asset | $	$ 258,621
Servicer advance investments | Minimum
Assets
Expected remaining life	1 year
Servicer advance investments | Maximum
Assets
Expected remaining life	2 years
Servicer advance investments | Weighted Average
Assets
Expected remaining life	2 years
Servicer advance investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.08
Servicer advance investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.14
Servicer advance investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.13
Servicer advance investments | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.03
Servicer advance investments | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.04
Servicer advance investments | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.04
Loan purchase and interest rate lock commitments
Liabilities
Loan purchase commitments, net | $	$ 10,282
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.15
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.15
Loan purchase and interest rate lock commitments | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.15
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate | Minimum
Assets
Loans receivable, measurement input	0.0350
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate | Maximum
Assets
Loans receivable, measurement input	0.0350
Loan purchase and interest rate lock commitments | Whole loan spread to swap rate | Weighted Average
Assets
Loans receivable, measurement input	0.0350
Loan purchase and interest rate lock commitments | Mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0.0006
Loan purchase and interest rate lock commitments | Mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0016
Loan purchase and interest rate lock commitments | Mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0008
Loan purchase and interest rate lock commitments | Committed sales price | Minimum
Liabilities
Loan purchase commitments, measurement input	100.89
Loan purchase and interest rate lock commitments | Committed sales price | Maximum
Liabilities
Loan purchase commitments, measurement input	103.40
Loan purchase and interest rate lock commitments | Committed sales price | Weighted Average
Liabilities
Loan purchase commitments, measurement input	102.59
Loan purchase and interest rate lock commitments | Pull-through rate | Minimum
Liabilities
Loan purchase commitments, measurement input	0.13
Loan purchase and interest rate lock commitments | Pull-through rate | Maximum
Liabilities
Loan purchase commitments, measurement input	1
Loan purchase and interest rate lock commitments | Pull-through rate | Weighted Average
Liabilities
Loan purchase commitments, measurement input	0.58
Loan purchase and interest rate lock commitments | MSR Multiple | Minimum
Assets
Loans receivable, measurement input	0.8
Loan purchase and interest rate lock commitments | MSR Multiple | Maximum
Assets
Loans receivable, measurement input	4.1
Loan purchase and interest rate lock commitments | MSR Multiple | Weighted Average
Assets
Loans receivable, measurement input	3.4
Loan purchase and interest rate lock commitments | Whole Loan Spread to TBA Price | Minimum
Assets
Loans receivable, measurement input	2.00
Loan purchase and interest rate lock commitments | Whole Loan Spread to TBA Price | Maximum
Assets
Loans receivable, measurement input	2.00
Loan purchase and interest rate lock commitments | Whole Loan Spread to TBA Price | Weighted Average
Assets
Loans receivable, measurement input	2.00
MSRs
Assets
Servicing asset | $	$ 14,878
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.08
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.97
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.26
MSRs | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.12
MSRs | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.12
MSRs | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.12
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing asset, measurement input	95
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing asset, measurement input	95
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing asset, measurement input	95
Excess MSRs
Assets
Servicing asset | $	$ 35,070
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.10
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.13
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.11
Excess MSRs | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.15
Excess MSRs | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.21
Excess MSRs | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Excess mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0.0009
Excess MSRs | Excess mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0017
Excess MSRs | Excess mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0012
Shared home appreciation options
Assets
Shared home appreciation options | $	$ 41,758
Shared home appreciation options | Prepayment rate (annual CPR) | Minimum
Assets
Shared home appreciation options			0.08
Shared home appreciation options | Prepayment rate (annual CPR) | Maximum
Assets
Shared home appreciation options			0.26
Shared home appreciation options | Prepayment rate (annual CPR) | Weighted Average
Assets
Shared home appreciation options			0.19
Shared home appreciation options | Discount rate | Minimum
Assets
Shared home appreciation options	0.16
Shared home appreciation options | Discount rate | Maximum
Assets
Shared home appreciation options	0.16
Shared home appreciation options | Discount rate | Weighted Average
Assets
Shared home appreciation options	0.16
Shared home appreciation options | Home price appreciation | Minimum
Assets
Shared home appreciation options			0.02
Shared home appreciation options | Home price appreciation | Maximum
Assets
Shared home appreciation options			0.03
Shared home appreciation options | Home price appreciation | Weighted Average
Assets
Shared home appreciation options			0.03
Guarantee asset
Assets
Guarantee asset | $	$ 579
Guarantee asset | Prepayment rate (annual CPR) | Minimum
Assets
Guarantee asset	0.42
Guarantee asset | Prepayment rate (annual CPR) | Maximum
Assets
Guarantee asset	0.42
Guarantee asset | Prepayment rate (annual CPR) | Weighted Average
Assets
Guarantee asset	0.42
Guarantee asset | Discount rate | Minimum
Assets
Guarantee asset	0.12
Guarantee asset | Discount rate | Maximum
Assets
Guarantee asset	0.12
Guarantee asset | Discount rate | Weighted Average
Assets
Guarantee asset	0.12
REO
Assets
REO | $	$ 3,523
REO | Loss severity | Minimum
Assets
REO, measurement input	0.03
REO | Loss severity | Maximum
Assets
REO, measurement input	0.63
REO | Loss severity | Weighted Average
Assets
REO, measurement input	0.23