Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2020 USD ($) $ / loan	Dec. 31, 2019
Loan purchase and interest rate lock commitments
Liabilities
Loan purchase commitments, net	$ 177
Loan purchase and interest rate lock commitments | Committed sales price | Minimum
Liabilities
Loan purchase commitments, measurement input	96.10
Loan purchase and interest rate lock commitments | Committed sales price | Maximum
Liabilities
Loan purchase commitments, measurement input	101.28
Loan purchase and interest rate lock commitments | Committed sales price | Weighted Average
Liabilities
Loan purchase commitments, measurement input	100.11
Loan purchase and interest rate lock commitments | Pull-through rate | Minimum
Liabilities
Loan purchase commitments, measurement input	1
Loan purchase and interest rate lock commitments | Pull-through rate | Maximum
Liabilities
Loan purchase commitments, measurement input	1
Loan purchase and interest rate lock commitments | Pull-through rate | Weighted Average
Liabilities
Loan purchase commitments, measurement input	1
ABS Issued
Liabilities
ABS issued	$ 2,162,134
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.10
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.24
ABS Issued | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
ABS Issued | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.25
ABS Issued | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.04
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.40
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.01
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.32
Freddie Mac SLST
Liabilities
ABS issued	$ 1,812,008
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
Freddie Mac SLST | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.14
Freddie Mac SLST | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.04
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.17
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.18
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.17
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.30
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
Freddie Mac K-Series
Liabilities
ABS issued	$ 464,691
Freddie Mac K-Series | IO prepayment rate (annual CPY/CPP) | Minimum
Liabilities
ABS issued, measurement input	1
Freddie Mac K-Series | IO prepayment rate (annual CPY/CPP) | Maximum
Liabilities
ABS issued, measurement input	1
Freddie Mac K-Series | IO prepayment rate (annual CPY/CPP) | Weighted Average
Liabilities
ABS issued, measurement input	1
Freddie Mac K-Series | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.19
Freddie Mac K-Series | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.02
Freddie Mac K-Series | Non-IO prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
Freddie Mac K-Series | Non-IO prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0
Freddie Mac K-Series | Non-IO prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0
CAFL
Liabilities
ABS issued	$ 2,417,253
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0
CAFL | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.01
CAFL | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.74
CAFL | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.04
Jumbo loans committed to sell
Assets
Loans receivable, fair value	$ 20,199
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Loans held-for-sale, measurement input | $ / loan	101.00
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Loans held-for-sale, measurement input | $ / loan	101.00
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Loans held-for-sale, measurement input | $ / loan	101.00
Legacy Sequoia
Assets
Loans receivable, fair value	$ 304,632
Sequoia Choice
Assets
Loans receivable, fair value	2,064,388
Liabilities
Fair value of securities owned	208,000
Freddie Mac SLST
Assets
Loans receivable, fair value	2,145,111
Liabilities
Fair value of securities owned	334,000
Single-family rental loans
Assets
Loans receivable, fair value	$ 379,795
Single-family rental loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.03
Single-family rental loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.31
Single-family rental loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.36
Single-family rental loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.34
Single-family rental loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.10
Single-family rental loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.10
Single-family rental loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.10
CAFL
Assets
Loans receivable, fair value	$ 2,615,038
Liabilities
Fair value of securities owned	203,000
Residential bridge loans
Assets
Loans receivable, fair value	$ 787,367
Residential bridge loans | Discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.08
Residential bridge loans | Discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.17
Residential bridge loans | Discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.11
Freddie Mac K-Series
Assets
Loans receivable, fair value	$ 489,075
Liabilities
Fair value of securities owned	$ 24,000
Business purpose residential loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input | $ / loan	0.0230
Business purpose residential loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input | $ / loan	0.0230
Business purpose residential loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input | $ / loan	0.0230
Business purpose residential loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input | $ / loan	0.0300
Business purpose residential loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input | $ / loan	0.2450
Business purpose residential loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input | $ / loan	0.0772
Loan purchase and interest rate lock commitments | Mortgage servicing income | Minimum
Assets
Servicing asset, measurement input | $ / loan	0.0008
Loan purchase and interest rate lock commitments | Mortgage servicing income | Maximum
Assets
Servicing asset, measurement input | $ / loan	0.0013
Loan purchase and interest rate lock commitments | Mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input | $ / loan	0.0010
Trading and AFS securities
Assets
Trading and AFS securities	$ 316,436
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0.06
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.65
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.17
Trading and AFS securities | Discount rate | Minimum
Assets
Trading and AFS securities	0.04
Trading and AFS securities | Discount rate | Maximum
Assets
Trading and AFS securities	0.21
Trading and AFS securities | Discount rate | Weighted Average
Assets
Trading and AFS securities	0.10
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.15
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.01
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.16
Servicer advance investments
Assets
Servicing asset	$ 266,948
Servicer advance investments | Minimum
Assets
Expected remaining life	2 years
Servicer advance investments | Maximum
Assets
Expected remaining life	2 years
Servicer advance investments | Weighted Average
Assets
Expected remaining life	2 years
Servicer advance investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.08
Servicer advance investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.14
Servicer advance investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.14
Servicer advance investments | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.05
Servicer advance investments | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.05
Servicer advance investments | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.05
MSRs
Assets
Servicing asset	$ 19,661
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.08
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.62
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.21
MSRs | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.12
MSRs | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.12
MSRs | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.12
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing asset, measurement input | $ / loan	95
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing asset, measurement input | $ / loan	95
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing asset, measurement input | $ / loan	95
Excess MSRs
Assets
Servicing asset	$ 36,197
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.10
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.14
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.12
Excess MSRs | Discount rate | Minimum
Assets
Servicing asset, measurement input	0.15
Excess MSRs | Discount rate | Maximum
Assets
Servicing asset, measurement input	0.20
Excess MSRs | Discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Excess mortgage servicing income | Minimum
Assets
Servicing asset, measurement input | $ / loan	0.0009
Excess MSRs | Excess mortgage servicing income | Maximum
Assets
Servicing asset, measurement input | $ / loan	0.0017
Excess MSRs | Excess mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input | $ / loan	0.0012
Shared home appreciation options
Assets
Shared home appreciation options	$ 40,851
Shared home appreciation options | Prepayment rate (annual CPR) | Minimum
Assets
Shared home appreciation options		0.08
Shared home appreciation options | Prepayment rate (annual CPR) | Maximum
Assets
Shared home appreciation options		0.28
Shared home appreciation options | Prepayment rate (annual CPR) | Weighted Average
Assets
Shared home appreciation options		0.21
Shared home appreciation options | Discount rate | Minimum
Assets
Shared home appreciation options	0.17
Shared home appreciation options | Discount rate | Maximum
Assets
Shared home appreciation options	0.17
Shared home appreciation options | Discount rate | Weighted Average
Assets
Shared home appreciation options	0.17
Shared home appreciation options | Home price appreciation | Minimum
Assets
Shared home appreciation options		0.01
Shared home appreciation options | Home price appreciation | Maximum
Assets
Shared home appreciation options		0.03
Shared home appreciation options | Home price appreciation | Weighted Average
Assets
Shared home appreciation options		0.03
Guarantee asset
Assets
Guarantee asset	$ 781
Guarantee asset | Prepayment rate (annual CPR) | Minimum
Assets
Guarantee asset	0.37
Guarantee asset | Prepayment rate (annual CPR) | Maximum
Assets
Guarantee asset	0.37
Guarantee asset | Prepayment rate (annual CPR) | Weighted Average
Assets
Guarantee asset	0.37
Guarantee asset | Discount rate | Minimum
Assets
Guarantee asset	0.13
Guarantee asset | Discount rate | Maximum
Assets
Guarantee asset	0.13
Guarantee asset | Discount rate | Weighted Average
Assets
Guarantee asset	0.13
REO
Assets
REO	$ 1,712
REO | Loss severity | Minimum
Assets
REO, measurement input	0.03
REO | Loss severity | Maximum
Assets
REO, measurement input	0.86
REO | Loss severity | Weighted Average
Assets
REO, measurement input	0.19