Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	$ 14,353	$ (3,922)	$ (4,261)
Total Market Valuation Gains (Losses), Net	(12,917)	(51,975)	(298)
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	36,445	6,482	30,985
Mortgage Banking Activities, Net | Residential Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	5,786	3,712	51,312
Mortgage Banking Activities, Net | Residential loan purchase and forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	25,613	50,234	13,891
Mortgage Banking Activities, Net | Commercial loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	433	10,265	20,788
Mortgage Banking Activities, Net | Net investments in consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	1,455	(15,261)	(23,839)
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	3,158	(42,468)	(31,167)
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(28,574)	(21,357)	(10,202)
Investment Fair Value Changes, Net | Net investments in consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(4,200)	(1,192)	(894)
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(9,112)	(9,677)	(7,792)
Investment Fair Value Changes, Net | Residential loans, held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(23,102)	(6,337)	(697)
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	9,666	(2,019)	(358)
Investment Fair Value Changes, Net | Valuation adjustments on commercial loans held-for-sale
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(307)	0	0
Investment Fair Value Changes, Net | Risk sharing investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,151)	(1,886)	104
Investment Fair Value Changes, Net | Impairments on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(368)	(246)	(565)
MSR Income (Loss), Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	(20,788)	(37,100)	(21,081)
MSR Income (Loss), Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	15,584	(12,708)	0
MSR Income (Loss), Net | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income (loss), net	$ (36,372)	$ (24,392)	$ (21,081)