Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	$ 7,860	$ 14,353	$ (3,922)
Total Market Valuation Gains (Losses), Net	51,484	(12,917)	(51,975)
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	51,844	36,445	6,482
Mortgage Banking Activities, Net | Residential Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	31,493	5,786	3,712
Mortgage Banking Activities, Net | Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	37,880	25,613	50,234
Mortgage Banking Activities, Net | Commercial loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	433	10,265
Mortgage Banking Activities, Net | Sequoia securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	0	1,455	(15,261)
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(17,529)	3,158	(42,468)
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	10,374	(28,574)	(21,357)
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(12,842)	(9,112)	(9,677)
Investment Fair Value Changes, Net | Residential loans, held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(5,765)	(23,102)	(6,337)
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	39,526	9,666	(2,019)
Investment Fair Value Changes, Net | Valuation adjustments on commercial loans held-for-sale
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	300	(307)	0
Investment Fair Value Changes, Net | Legacy Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(8,027)	(4,200)	(1,192)
Investment Fair Value Changes, Net | Sequoia Choice
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(323)	0	0
Investment Fair Value Changes, Net | Risk-sharing investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,484)	(1,151)	(1,886)
Investment Fair Value Changes, Net | Impairments on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,011)	(368)	(246)
MSR Income (Loss), Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	(10,734)	(20,788)	(37,100)
MSR Income (Loss), Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	(568)	15,584	(12,708)
MSR Income (Loss), Net | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	$ (10,166)	$ (36,372)	$ (24,392)