Fair Value Measurements - Gains (Losses) Recognized on Assets Measured at Fair Value on a Non-recurring Basis (Details) - Measured on a non-recurring basis - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Gains (losses) recognized on assets measured at fair value on a non-recurring basis	$ (4,985)	$ (7,905)	$ (20,899)
Non-performing loans held-for-sale, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Gains (losses) recognized on assets measured at fair value on a non-recurring basis	(445)	0	(520)
Impaired loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Gains (losses) recognized on assets measured at fair value on a non-recurring basis	(4,284)	(6,311)	(21,992)
MSR, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Gains (losses) recognized on assets measured at fair value on a non-recurring basis	568	60	5,401
REO, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Gains (losses) recognized on assets measured at fair value on a non-recurring basis	$ (824)	$ (1,654)	$ (3,788)