Fair Value Measurements - Schedule of Carrying Values of Assets Measured at Fair Value (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	$ 280,045	$ 416,798
Impaired loans	213,663	237,518
MSR, net	10,130	11,014
REO, net	15,144	19,798
Carrying value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	11,584	8,960
MSR, net	10,130	11,014
Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	0	0
MSR, net	0	0
Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	11,589	9,037
MSR, net	10,133	11,017
Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	280,045	416,798
Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	4	2
Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	280,041	416,796
Non-recurring basis | Level 3 | Carrying value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impaired loans	124,101	134,910
MSR, net	10,130	11,014
REO, net	14,428	16,307
Total	148,802	163,813
Non-recurring basis | Level 3 | Non-performing loans held-for-sale, net | Carrying value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Non-performing loans held-for-sale, net	143	1,582
GSE issuance REMICs and CMOs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	240,793	330,539
GSE issuance REMICs and CMOs | Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	240,793	330,539
GSE issuance REMICs and CMOs | Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	0	0
GSE issuance REMICs and CMOs | Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	240,793	330,539
Non-GSE issuance REMICs and CMOs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	1,443	3,054
Non-GSE issuance REMICs and CMOs | Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	1,443	3,054
Non-GSE issuance REMICs and CMOs | Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	0	0
Non-GSE issuance REMICs and CMOs | Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	1,443	3,054
GSE pass-through certificates
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	8,930	11,264
GSE pass-through certificates | Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	8,930	11,264
GSE pass-through certificates | Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	0	0
GSE pass-through certificates | Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	8,930	11,264
Obligations of GSEs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	28,875	71,939
Obligations of GSEs | Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	28,875	71,939
Obligations of GSEs | Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	0	0
Obligations of GSEs | Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	28,875	71,939
Fannie Mae stock
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	4	2
Fannie Mae stock | Recurring basis
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	4	2
Fannie Mae stock | Recurring basis | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	4	2
Fannie Mae stock | Recurring basis | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total securities available-for-sale	$ 0	$ 0