Loans Receivable and Allowance for Loan Losses - Schedule of Average Recorded Investment, Interest Income Recognized and Cash Basis Interest Related to Impaired Loans (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Without an allowance recorded:
Average Recorded Investment	$ 226,361	$ 242,980	$ 330,190
Interest Income Recognized	7,218	8,765	10,714
Cash Basis Interest Income	7,309	8,865	10,922
Consumer and other loans (gross) | Home equity lines of credit
With an allowance recorded:
Average Recorded Investment	4,445	5,585	5,202
Interest Income Recognized	124	48	45
Cash Basis Interest Income	126	54	54
Consumer and other loans (gross) | Commercial and industrial
Without an allowance recorded:
Average Recorded Investment	65	0	0
Interest Income Recognized	2	0	0
Cash Basis Interest Income	2	0	0
Mortgage Loans | Residential | Full documentation interest-only
With an allowance recorded:
Average Recorded Investment	24,322	39,506	84,264
Interest Income Recognized	364	860	1,860
Cash Basis Interest Income	380	877	1,920
Mortgage Loans | Residential | Full documentation amortizing
With an allowance recorded:
Average Recorded Investment	72,452	52,426	38,340
Interest Income Recognized	2,359	1,893	1,491
Cash Basis Interest Income	2,388	1,920	1,498
Without an allowance recorded:
Average Recorded Investment	0	0	365
Interest Income Recognized	0	0	0
Cash Basis Interest Income	0	0	0
Mortgage Loans | Residential | Reduced documentation interest-only
With an allowance recorded:
Average Recorded Investment	34,171	66,321	112,172
Interest Income Recognized	861	1,910	3,646
Cash Basis Interest Income	851	1,923	3,671
Without an allowance recorded:
Average Recorded Investment	519	0	0
Interest Income Recognized	79	0	0
Cash Basis Interest Income	78	0	0
Mortgage Loans | Residential | Reduced documentation amortizing
With an allowance recorded:
Average Recorded Investment	64,230	30,310	22,137
Interest Income Recognized	2,454	1,927	655
Cash Basis Interest Income	2,472	1,936	653
Mortgage Loans | Multi-family
With an allowance recorded:
Average Recorded Investment	4,845	14,390	30,291
Interest Income Recognized	145	415	1,320
Cash Basis Interest Income	149	417	1,339
Without an allowance recorded:
Average Recorded Investment	9,246	16,935	17,225
Interest Income Recognized	249	857	632
Cash Basis Interest Income	262	862	633
Mortgage Loans | Commercial real estate
With an allowance recorded:
Average Recorded Investment	1,673	11,875	17,341
Interest Income Recognized	0	333	1,065
Cash Basis Interest Income	0	348	1,154
Without an allowance recorded:
Average Recorded Investment	10,393	5,632	2,853
Interest Income Recognized	581	522	0
Cash Basis Interest Income	$ 601	$ 528	$ 0