Fair Value Measurements - Schedule of Carrying Values of Assets Measured at Fair Value on a Non-Recurring Basis (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Fair Value of Financial Instruments
Impaired loans	$ 212,371	$ 213,663
MSR, net	10,237	10,130
REO, net	13,500	15,144
Carrying Value
Fair Value of Financial Instruments
Non-performing loans held-for-sale, net	6,236	11,584
MSR, net	10,237	10,130
Level 3
Fair Value of Financial Instruments
Non-performing loans held-for-sale, net	6,360	11,589
MSR, net	10,240	10,133
Nonrecurring | Level 3 | Carrying Value
Fair Value of Financial Instruments
Non-performing loans held-for-sale, net	141	143
Impaired loans	124,766	124,101
MSR, net	10,237	10,130
REO, net	13,500	14,428
Total	$ 148,644	$ 148,802