Loans and Credit Quality - Impaired Loans (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015	Dec. 31, 2015
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	$ 40,793		$ 40,793		$ 33,579
Impaired loan, no related allowance, unpaid principal balance	41,496		41,496		33,992
Impaired loan, related allowance, recorded investment	3,972		3,972		1,061
Impaired loan, related allowance, unpaid principal balance	3,972		3,972		1,061
Related allowance	535		535		344
Recorded investment	44,765		44,765		34,640
Unpaid principal balance	45,468		45,468		35,053
Impaired loan, no related allowance, average recorded investment	40,927	$ 27,763	40,964	$ 21,176
Impaired loan, no related allowance, interest income on accrual method	981	308	1,922	644
Impaired loan, related allowance, average recorded investment	3,972	984	2,947	1,970
Impaired loan, related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	44,899	28,747	43,911	23,146
Impaired loan, interest income on accrual method	981	308	1,922	644
Commercial | Asset Quality Rating - Special Mention
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	150		150		157
Impaired loan, no related allowance, unpaid principal balance	150		150		157
Related allowance	0		0		0
Recorded investment	150		150		157
Unpaid principal balance	150		150		157
Impaired loan, no related allowance, average recorded investment	150	161	153	165
Impaired loan, no related allowance, interest income on accrual method	6	3	12	10
Impaired loan, average recorded investment	150	161	153	165
Impaired loan, interest income on accrual method	6	3	12	10
Commercial | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	19,623		19,623		14,030
Impaired loan, no related allowance, unpaid principal balance	20,303		20,303		14,443
Impaired loan, related allowance, recorded investment					1,061
Impaired loan, related allowance, unpaid principal balance					1,061
Related allowance	0		0		344
Recorded investment	19,623		19,623		15,091
Unpaid principal balance	20,303		20,303		15,504
Impaired loan, no related allowance, average recorded investment	19,681	13,661	18,660	9,711
Impaired loan, no related allowance, interest income on accrual method	428	155	801	225
Impaired loan, related allowance, average recorded investment	0	944	197	1,863
Impaired loan, related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	19,681	14,605	18,857	11,574
Impaired loan, interest income on accrual method	428	155	801	225
Real estate construction one-to-four family | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, related allowance, recorded investment	3,972		3,972
Impaired loan, related allowance, unpaid principal balance	3,972		3,972
Related allowance	535		535
Recorded investment	3,972		3,972
Unpaid principal balance	3,972		3,972
Impaired loan, no related allowance, average recorded investment			1,319	0
Impaired loan, no related allowance, interest income on accrual method			0	0
Impaired loan, related allowance, average recorded investment	3,972	0	2,653	0
Impaired loan, related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	3,972	0	3,972	0
Impaired loan, interest income on accrual method	0	0	0	0
Real estate construction other | Asset Quality Rating - Pass
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, average recorded investment	0	726	0	744
Impaired loan, no related allowance, interest income on accrual method	0	13	0	72
Impaired loan, average recorded investment	0	726	0	744
Impaired loan, interest income on accrual method	0	13	0	72
Real estate construction other | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	1,912		1,912
Impaired loan, no related allowance, unpaid principal balance	1,912		1,912
Related allowance	0		0
Recorded investment	1,912		1,912
Unpaid principal balance	1,912		1,912
Impaired loan, no related allowance, average recorded investment	1,912	5,678	1,912	1,913
Impaired loan, no related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	1,912	5,678	1,912	1,913
Impaired loan, interest income on accrual method	0	0	0	0
Real estate term owner occupied | Asset Quality Rating - Pass
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	254		254		753
Impaired loan, no related allowance, unpaid principal balance	254		254		753
Related allowance	0		0		0
Recorded investment	254		254		753
Unpaid principal balance	254		254		753
Impaired loan, no related allowance, average recorded investment	256	761	420	677
Impaired loan, no related allowance, interest income on accrual method	10	17	29	46
Impaired loan, average recorded investment	256	761	420	677
Impaired loan, interest income on accrual method	10	17	29	46
Real estate term owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, average recorded investment			0	90
Impaired loan, no related allowance, interest income on accrual method			0	5
Impaired loan, average recorded investment			0	90
Impaired loan, interest income on accrual method			0	5
Real estate term owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	16,765		16,765		16,476
Impaired loan, no related allowance, unpaid principal balance	16,765		16,765		16,476
Related allowance	0		0		0
Recorded investment	16,765		16,765		16,476
Unpaid principal balance	16,765		16,765		16,476
Impaired loan, no related allowance, average recorded investment	16,787	5,203	16,491	3,720
Impaired loan, no related allowance, interest income on accrual method	466	85	934	113
Impaired loan, average recorded investment	16,787	5,203	16,491	3,720
Impaired loan, interest income on accrual method	466	85	934	113
Real estate term non-owner occupied | Asset Quality Rating - Pass
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	412		412		473
Impaired loan, no related allowance, unpaid principal balance	412		412		473
Related allowance	0		0		0
Recorded investment	412		412		473
Unpaid principal balance	412		412		473
Impaired loan, no related allowance, average recorded investment	434	516	455	534
Impaired loan, no related allowance, interest income on accrual method	36	19	73	57
Impaired loan, average recorded investment	434	516	455	534
Impaired loan, interest income on accrual method	36	19	73	57
Real estate term non-owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, average recorded investment	0	101	0	1,444
Impaired loan, no related allowance, interest income on accrual method	0	10	0	97
Impaired loan, average recorded investment	0	101	0	1,444
Impaired loan, interest income on accrual method	0	10	0	97
Real estate term non-owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	204		204		352
Impaired loan, no related allowance, unpaid principal balance	204		204		352
Related allowance	0		0		0
Recorded investment	204		204		352
Unpaid principal balance	204		204		352
Impaired loan, no related allowance, average recorded investment	210	269	223	1,423
Impaired loan, no related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	210	269	223	1,423
Impaired loan, interest income on accrual method	0	0	0	0
Real estate term other | Asset Quality Rating - Pass
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	644		644		699
Impaired loan, no related allowance, unpaid principal balance	644		644		699
Related allowance	0		0		0
Recorded investment	644		644		699
Unpaid principal balance	644		644		699
Impaired loan, no related allowance, average recorded investment	653	0	623	0
Impaired loan, no related allowance, interest income on accrual method	23	0	47	0
Impaired loan, average recorded investment	653	0	623	0
Impaired loan, interest income on accrual method	23	0	47	0
Real estate term other | Asset Quality Rating - Special Mention
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment					91
Impaired loan, no related allowance, unpaid principal balance					91
Related allowance					0
Recorded investment					91
Unpaid principal balance					91
Impaired loan, no related allowance, average recorded investment	0	148	58	50
Impaired loan, no related allowance, interest income on accrual method	0	3	4	3
Impaired loan, average recorded investment	0	148	58	50
Impaired loan, interest income on accrual method	0	3	4	3
Real estate term other | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, average recorded investment			0	99
Impaired loan, no related allowance, interest income on accrual method			0	7
Impaired loan, related allowance, average recorded investment			0	93
Impaired loan, related allowance, interest income on accrual method			0	0
Impaired loan, average recorded investment			0	192
Impaired loan, interest income on accrual method			0	7
Consumer secured by 1st deeds of trust | Asset Quality Rating - Pass
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment					76
Impaired loan, no related allowance, unpaid principal balance					76
Related allowance					0
Recorded investment					76
Unpaid principal balance					76
Impaired loan, no related allowance, average recorded investment			50	80
Impaired loan, no related allowance, interest income on accrual method			2	3
Impaired loan, average recorded investment			50	80
Impaired loan, interest income on accrual method			2	3
Consumer secured by 1st deeds of trust | Asset Quality Rating - Special Mention
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, average recorded investment	0	78
Impaired loan, no related allowance, interest income on accrual method	0	1
Impaired loan, average recorded investment	0	78
Impaired loan, interest income on accrual method	0	1
Consumer secured by 1st deeds of trust | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, no related allowance, recorded investment	829		829		472
Impaired loan, no related allowance, unpaid principal balance	852		852		472
Related allowance	0		0		0
Recorded investment	829		829		472
Unpaid principal balance	852		852		$ 472
Impaired loan, no related allowance, average recorded investment	844	461	600	526
Impaired loan, no related allowance, interest income on accrual method	12	2	20	6
Impaired loan, related allowance, average recorded investment			97	0
Impaired loan, related allowance, interest income on accrual method			0	0
Impaired loan, average recorded investment	844	461	697	526
Impaired loan, interest income on accrual method	12	2	20	6
Consumer other | Asset Quality Rating - Substandard
Financing Receivable, Impaired [Line Items]
Impaired loan, related allowance, average recorded investment	0	40	0	14
Impaired loan, related allowance, interest income on accrual method	0	0	0	0
Impaired loan, average recorded investment	0	40	0	14
Impaired loan, interest income on accrual method	$ 0	$ 0	$ 0	$ 0