Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2019	Dec. 31, 2018
Financing Receivable, Past Due [Line Items]
Total Past Due	$ 7,702	$ 3,440
Nonaccrual	18,080	15,210
Current	994,240	970,183
Total financing receivables	1,020,022	988,833
Less: Unearned origination fees, net of origination costs	(4,318)	(4,487)
Total loans	1,015,704	984,346
30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	6,049	2,483
Nonaccrual	2,093	1,329
60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	1,653	957
Nonaccrual	0	324
Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	6,983	3,597
Commercial
Financing Receivable, Past Due [Line Items]
Total Past Due	6,273	1,729
Nonaccrual	11,207	12,671
Current	369,777	328,020
Total financing receivables	387,257	342,420
Total loans	385,953	341,091
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	4,800	872
Nonaccrual	385	1,329
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	1,473	857
Nonaccrual	0	324
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	3,745	1,287
Real estate construction one-to-four family
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,492	0
Current	37,521	37,111
Total financing receivables	39,013	37,111
Total loans	38,851	36,828
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0
Real estate construction other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	58,824	72,256
Total financing receivables	58,824	72,256
Total loans	58,380	71,658
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	987	1,197
Nonaccrual	3,844	1,694
Current	122,160	123,523
Total financing receivables	126,991	126,414
Total loans	126,360	125,795
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	987	1,197
Nonaccrual	1,087	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	2,568	1,694
Real estate term non-owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	325,738	325,720
Total financing receivables	325,738	325,720
Total loans	324,104	324,198
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,198	577
Current	40,767	41,462
Total financing receivables	41,965	42,039
Total loans	41,658	41,746
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	621	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	577	577
Consumer secured by 1st deeds of trust
Financing Receivable, Past Due [Line Items]
Total Past Due	209	324
Nonaccrual	210	220
Current	16,429	18,684
Total financing receivables	16,848	19,228
Total loans	16,848	19,234
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	29	224
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	180	100
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer other
Financing Receivable, Past Due [Line Items]
Total Past Due	233	190
Nonaccrual	129	48
Current	23,024	23,407
Total financing receivables	23,386	23,645
Total loans	23,550	23,796
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	233	190
Nonaccrual	0	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	$ 93	$ 39