Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	9 Months Ended
	Sep. 30, 2019 USD ($) $ / loan	Dec. 31, 2018 USD ($)
Liabilities
Contingent consideration | $	$ 25,167	$ 0
ABS Issued
Liabilities
ABS issued | $	$ 2,781,001
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.08
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.40
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.20
ABS Issued | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.03
ABS Issued | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.15
ABS Issued | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.04
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.07
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.02
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.20
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.29
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.21
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,987,473
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
Freddie Mac SLST | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.13
Freddie Mac SLST | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.03
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.22
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.22
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.22
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.30
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 3,577,577
Freddie Mac K-Series | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
Freddie Mac K-Series | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0
Freddie Mac K-Series | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0
Freddie Mac K-Series | Discount rate | Minimum
Liabilities
ABS issued, measurement input	0.02
Freddie Mac K-Series | Discount rate | Maximum
Liabilities
ABS issued, measurement input	0.09
Freddie Mac K-Series | Discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.02
Freddie Mac K-Series | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.20
Freddie Mac K-Series | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.20
Freddie Mac K-Series | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.20
Loan purchase commitments
Assets
Loan purchase commitments, net | $	$ 3,042
Loan purchase commitments | Prepayment rate (annual CPR) | Minimum
Assets
Residential loan purchase commitments, net	0.15
Loan purchase commitments | Prepayment rate (annual CPR) | Maximum
Assets
Residential loan purchase commitments, net	0.20
Loan purchase commitments | Prepayment rate (annual CPR) | Weighted Average
Assets
Residential loan purchase commitments, net	0.20
Loan purchase commitments | Whole loan spread to TBA price | Minimum
Assets
Residential loan purchase commitments, net	0.56
Loan purchase commitments | Whole loan spread to TBA price | Maximum
Assets
Residential loan purchase commitments, net	1.56
Loan purchase commitments | Whole loan spread to TBA price | Weighted Average
Assets
Residential loan purchase commitments, net	1.55
Loan purchase commitments | Mortgage servicing income | Minimum
Assets
Residential loan purchase commitments, net	0.0006
Loan purchase commitments | Mortgage servicing income | Maximum
Assets
Residential loan purchase commitments, net	0.0014
Loan purchase commitments | Mortgage servicing income | Weighted Average
Assets
Residential loan purchase commitments, net	0.001
Loan purchase commitments | MSR multiple | Minimum
Assets
Residential loan purchase commitments, net	0.6
Loan purchase commitments | MSR multiple | Maximum
Assets
Residential loan purchase commitments, net	4.6
Loan purchase commitments | MSR multiple | Weighted Average
Assets
Residential loan purchase commitments, net	2.5
Loan purchase commitments | Pull-through rate | Minimum
Assets
Residential loan purchase commitments, net	0.09
Loan purchase commitments | Pull-through rate | Maximum
Assets
Residential loan purchase commitments, net	1
Loan purchase commitments | Pull-through rate | Weighted Average
Assets
Residential loan purchase commitments, net	0.71
Loan purchase commitments | Whole loan spread to swap rate - fixed rate | Minimum
Assets
Residential loan purchase commitments, net	0.0115
Loan purchase commitments | Whole loan spread to swap rate - fixed rate | Maximum
Assets
Residential loan purchase commitments, net	0.0375
Loan purchase commitments | Whole loan spread to swap rate - fixed rate | Weighted Average
Assets
Residential loan purchase commitments, net	0.0257
Loan purchase commitments | Whole loan spread to swap rate - hybrid | Minimum
Assets
Residential loan purchase commitments, net	0.0090
Loan purchase commitments | Whole loan spread to swap rate - hybrid | Maximum
Assets
Residential loan purchase commitments, net	0.0330
Loan purchase commitments | Whole loan spread to swap rate - hybrid | Weighted Average
Assets
Residential loan purchase commitments, net	0.0128
Contingent consideration
Liabilities
Contingent consideration | $	$ 25,167
Contingent consideration | Discount rate | Minimum
Liabilities
Contingent consideration, measurement input	0.23
Contingent consideration | Discount rate | Maximum
Liabilities
Contingent consideration, measurement input	0.23
Contingent consideration | Discount rate | Weighted Average
Liabilities
Contingent consideration, measurement input	0.23
Contingent consideration | Probability of outcomes | Minimum
Liabilities
Contingent consideration, measurement input	0
Contingent consideration | Probability of outcomes | Maximum
Liabilities
Contingent consideration, measurement input	1
Contingent consideration | Probability of outcomes | Weighted Average
Liabilities
Contingent consideration, measurement input	0.90
Jumbo fixed-rate loans
Assets
Residential loans, at fair value | $	$ 2,452,300
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Minimum
Assets
Residential loans, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Maximum
Assets
Residential loans, measurement input	0.20
Jumbo fixed-rate loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Residential loans, measurement input	0.20
Jumbo fixed-rate loans | Whole loan spread to TBA price | Minimum
Assets
Residential loans, measurement input	0.56
Jumbo fixed-rate loans | Whole loan spread to TBA price | Maximum
Assets
Residential loans, measurement input	1.56
Jumbo fixed-rate loans | Whole loan spread to TBA price | Weighted Average
Assets
Residential loans, measurement input	1.55
Jumbo fixed-rate loans | Whole loan spread to swap rate | Minimum
Assets
Residential loans, measurement input	0.0094
Jumbo fixed-rate loans | Whole loan spread to swap rate | Maximum
Assets
Residential loans, measurement input	0.0375
Jumbo fixed-rate loans | Whole loan spread to swap rate | Weighted Average
Assets
Residential loans, measurement input	0.0184
Jumbo hybrid loans
Assets
Residential loans, at fair value | $	$ 321,793
Jumbo hybrid loans | Prepayment rate (annual CPR) | Minimum
Assets
Residential loans, measurement input	0.15
Jumbo hybrid loans | Prepayment rate (annual CPR) | Maximum
Assets
Residential loans, measurement input	0.15
Jumbo hybrid loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Residential loans, measurement input	0.15
Jumbo hybrid loans | Whole loan spread to swap rate | Minimum
Assets
Residential loans, measurement input	0.0090
Jumbo hybrid loans | Whole loan spread to swap rate | Maximum
Assets
Residential loans, measurement input	0.0345
Jumbo hybrid loans | Whole loan spread to swap rate | Weighted Average
Assets
Residential loans, measurement input	0.0146
Jumbo loans committed to sell
Assets
Residential loans, at fair value | $	$ 418,905
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Residential loans, measurement input	101.88
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Residential loans, measurement input	102.91
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Residential loans, measurement input	102.27
Legacy Sequoia
Assets
Residential loans, at fair value | $	$ 429,159
Liabilities
Fair value of securities owned | $	266,000
Sequoia Choice
Assets
Residential loans, at fair value | $	2,618,316
Liabilities
Fair value of securities owned | $	454,000
Freddie Mac SLST
Assets
Residential loans, at fair value | $	2,441,223
Liabilities
Fair value of securities owned | $	214,000
Business purpose residential loans, at fair value
Assets
Residential loans, at fair value | $	$ 129,145
Business purpose residential loans, at fair value | Prepayment rate (annual CPR) | Minimum
Assets
Residential loans, measurement input	0.01
Business purpose residential loans, at fair value | Prepayment rate (annual CPR) | Maximum
Assets
Residential loans, measurement input	0.10
Business purpose residential loans, at fair value | Prepayment rate (annual CPR) | Weighted Average
Assets
Residential loans, measurement input	0.05
Business purpose residential loans, at fair value | Discount rate | Minimum
Assets
Residential loans, measurement input	0.05
Business purpose residential loans, at fair value | Discount rate | Maximum
Assets
Residential loans, measurement input	0.08
Business purpose residential loans, at fair value | Discount rate | Weighted Average
Assets
Residential loans, measurement input	0.08
Business purpose residential loans, at fair value | Senior credit spread | Minimum
Assets
Residential loans, measurement input	0.0110
Business purpose residential loans, at fair value | Senior credit spread | Maximum
Assets
Residential loans, measurement input	0.0110
Business purpose residential loans, at fair value | Senior credit spread | Weighted Average
Assets
Residential loans, measurement input	0.0110
Business purpose residential loans, at fair value | Subordinate credit spread | Minimum
Assets
Residential loans, measurement input	0.0143
Business purpose residential loans, at fair value | Subordinate credit spread | Maximum
Assets
Residential loans, measurement input	0.1250
Business purpose residential loans, at fair value | Subordinate credit spread | Weighted Average
Assets
Residential loans, measurement input	0.0308
Business purpose residential loans, at fair value | Senior credit support | Minimum
Assets
MSR, measurement input	0.35
Business purpose residential loans, at fair value | Senior credit support | Maximum
Assets
MSR, measurement input	0.36
Business purpose residential loans, at fair value | Senior credit support | Weighted Average
Assets
MSR, measurement input	0.36
Residential bridge loans
Assets
Residential loans, at fair value | $	$ 206,890
Residential bridge loans | Discount rate | Minimum
Assets
Residential loans, measurement input	0.06
Residential bridge loans | Discount rate | Maximum
Assets
Residential loans, measurement input	0.10
Residential bridge loans | Discount rate | Weighted Average
Assets
Residential loans, measurement input	0.07
Freddie Mac K-Series
Assets
Residential loans, at fair value | $	$ 3,791,622
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 1,285,426
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.60
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.13
Trading and AFS securities | Discount rate | Minimum
Assets
Trading and AFS securities	0.02
Trading and AFS securities | Discount rate | Maximum
Assets
Trading and AFS securities	0.15
Trading and AFS securities | Discount rate | Weighted Average
Assets
Trading and AFS securities	0.05
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.20
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.01
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.40
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.21
Servicer advance investments
Assets
Mortgage servicing rights | $	$ 222,591
Servicer advance investments | Minimum
Assets
Expected remaining life	2 years
Servicer advance investments | Maximum
Assets
Expected remaining life	2 years
Servicer advance investments | Weighted Average
Assets
Expected remaining life	2 years
Servicer advance investments | Prepayment rate (annual CPR) | Minimum
Assets
MSR, measurement input	0.08
Servicer advance investments | Prepayment rate (annual CPR) | Maximum
Assets
MSR, measurement input	0.15
Servicer advance investments | Prepayment rate (annual CPR) | Weighted Average
Assets
MSR, measurement input	0.14
Servicer advance investments | Discount rate | Minimum
Assets
MSR, measurement input	0.05
Servicer advance investments | Discount rate | Maximum
Assets
MSR, measurement input	0.05
Servicer advance investments | Discount rate | Weighted Average
Assets
MSR, measurement input	0.05
MSRs
Assets
Mortgage servicing rights | $	$ 39,837
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
MSR, measurement input	0.06
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
MSR, measurement input	0.53
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
MSR, measurement input	0.14
MSRs | Discount rate | Minimum
Assets
MSR, measurement input	0.11
MSRs | Discount rate | Maximum
Assets
MSR, measurement input	0.13
MSRs | Discount rate | Weighted Average
Assets
MSR, measurement input	0.11
MSRs | Per loan annual cost to service | Minimum
Assets
MSR, measurement input	82
MSRs | Per loan annual cost to service | Maximum
Assets
MSR, measurement input	82
MSRs | Per loan annual cost to service | Weighted Average
Assets
MSR, measurement input	82
Excess MSRs
Assets
Mortgage servicing rights | $	$ 32,937
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
MSR, measurement input	0.09
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
MSR, measurement input	0.14
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
MSR, measurement input	0.11
Excess MSRs | Discount rate | Minimum
Assets
MSR, measurement input	0.11
Excess MSRs | Discount rate | Maximum
Assets
MSR, measurement input	0.16
Excess MSRs | Discount rate | Weighted Average
Assets
MSR, measurement input	0.14
Excess MSRs | Excess mortgage servicing income | Minimum
Assets
MSR, measurement input	0.0008
Excess MSRs | Excess mortgage servicing income | Maximum
Assets
MSR, measurement input	0.0017
Excess MSRs | Excess mortgage servicing income | Weighted Average
Assets
MSR, measurement input	0.0013
Home appreciation options
Assets
Shared home appreciation options | $	$ 11,372
Guarantee asset
Assets
Guarantee asset | $	$ 1,784
Guarantee asset | Prepayment rate (annual CPR) | Minimum
Assets
Guarantee asset	0.16
Shared home appreciation options	0.10
Guarantee asset | Prepayment rate (annual CPR) | Maximum
Assets
Guarantee asset	0.16
Shared home appreciation options	0.30
Guarantee asset | Prepayment rate (annual CPR) | Weighted Average
Assets
Guarantee asset	0.16
Shared home appreciation options	0.23
Guarantee asset | Discount rate | Minimum
Assets
Guarantee asset	0.11
Shared home appreciation options	0.11
Guarantee asset | Discount rate | Maximum
Assets
Guarantee asset	0.11
Shared home appreciation options	0.11
Guarantee asset | Discount rate | Weighted Average
Assets
Guarantee asset	0.11
Shared home appreciation options	0.11
Guarantee asset | Home price appreciation | Minimum
Assets
Shared home appreciation options	0.03
Guarantee asset | Home price appreciation | Maximum
Assets
Shared home appreciation options	0.03
Guarantee asset | Home price appreciation | Weighted Average
Assets
Shared home appreciation options	0.03
REO
Assets
REO | $	$ 4,525
REO | Loss severity | Minimum
Assets
REO, measurement input	0.16
REO | Loss severity | Maximum
Assets
REO, measurement input	0.16
REO | Loss severity | Weighted Average
Assets
REO, measurement input	0.16