Fair Value Measurements - Carrying Value of Assets Measured at Fair Value on a Non-recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impaired loans	$ 237,518	$ 248,436
MSR, net	11,014	11,401
REO, net	19,798	35,723
Carrying Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	8,960	7,640
MSR, net	11,014	11,401
Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	9,037	7,955
MSR, net	11,017	11,406
Measured on a non-recurring basis | Level 3 | Carrying Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impaired loans	134,910	140,663
MSR, net	11,014	11,401
REO, net	16,307	19,375
Total	163,813	171,592
Measured on a non-recurring basis | Level 3 | Carrying Value | Non-performing loans held-for-sale, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	$ 1,582	$ 153