Financing Receivables - Impaired Loans (Details 5) (USD $) In Thousands, unless otherwise specified	3 Months Ended	12 Months Ended
	Mar. 31, 2014	Dec. 31, 2013
Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment	14,929	16,470
Unpaid Principal Balance	14,929	16,470
Average Recorded Investment	15,693	18,539
Interest Income Recognized	182	825
Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		74
Interest Income Recognized
Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	10,461	10,827
Unpaid Principal Balance, with no related allowance	10,461	10,827
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	10,645	12,212
Interest Income Recognized, with no related allowance	161	650
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		7
Interest Income Recognized
Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	2,337	2,662
Unpaid Principal Balance, with no related allowance	2,337	2,662
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	2,492	2,213
Interest Income Recognized, with no related allowance	9	108
Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	2,131	2,981
Unpaid Principal Balance, with no related allowance	2,131	2,981
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	2,556	3,479
Interest Income Recognized, with no related allowance	12	67
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		554
Interest Income Recognized
1-4 Family Residential | Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment	6,616	8,033
Unpaid Principal Balance	6,616	8,033
Average Recorded Investment	7,325	9,142
Interest Income Recognized	58	331
1-4 Family Residential | Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		74
Interest Income Recognized
1-4 Family Residential | Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	2,905	3,180
Unpaid Principal Balance, with no related allowance	2,905	3,180
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	3,043	3,556
Interest Income Recognized, with no related allowance	44	192
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		7
Interest Income Recognized
1-4 Family Residential | Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	2,314	2,646
Unpaid Principal Balance, with no related allowance	2,314	2,646
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	2,480	2,185
Interest Income Recognized, with no related allowance	9	107
1-4 Family Residential | Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	1,397	2,207
Unpaid Principal Balance, with no related allowance	1,397	2,207
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	1,802	2,847
Interest Income Recognized, with no related allowance	5	32
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		473
Interest Income Recognized
Multi-family | Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment	1,150	1,151
Unpaid Principal Balance	1,150	1,151
Average Recorded Investment	1,151	1,154
Interest Income Recognized	6	22
Multi-family | Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Multi-family | Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	1,055	1,056
Unpaid Principal Balance, with no related allowance	1,055	1,056
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	1,056	1,059
Interest Income Recognized, with no related allowance	6	22
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Multi-family | Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Multi-family | Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	95	95
Unpaid Principal Balance, with no related allowance	95	95
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	95	95
Interest Income Recognized, with no related allowance
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Construction | Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment
Unpaid Principal Balance
Average Recorded Investment
Interest Income Recognized
Construction | Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Construction | Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Construction | Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Construction | Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Commercial | Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment	7,140	7,270
Unpaid Principal Balance	7,140	7,270
Average Recorded Investment	7,205	8,174
Interest Income Recognized	118	471
Commercial | Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Commercial | Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	6,501	6,591
Unpaid Principal Balance, with no related allowance	6,501	6,591
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	6,546	7,556
Interest Income Recognized, with no related allowance	111	436
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Commercial | Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Commercial | Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	639	679
Unpaid Principal Balance, with no related allowance	639	679
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	659	537
Interest Income Recognized, with no related allowance	7	35
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment		81
Interest Income Recognized
Consumer | Loans Receivable
Financing Receivable, Impaired [Line Items]
Related Allowance
Recorded Investment	23	16
Unpaid Principal Balance	23	16
Average Recorded Investment	12	69
Interest Income Recognized		1
Consumer | Purchased loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Consumer | Purchased loans | With fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance		41
Interest Income Recognized, with no related allowance
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized
Consumer | Purchased loans | With no fair value discount
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance	23	16
Unpaid Principal Balance, with no related allowance	23	16
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance	12	28
Interest Income Recognized, with no related allowance		1
Consumer | Originated loans
Financing Receivable, Impaired [Line Items]
Recorded Investment, with no related allowance
Unpaid Principal Balance, with no related allowance
Related Allowance, with no related allowance
Average Recorded Investment, with no related allowance
Interest Income Recognized, with no related allowance
Recorded Investment, with related allowance
Unpaid Principal Balance, with related allowance
Related Allowance
Average Recorded Investment
Interest Income Recognized