Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Sep. 30, 2019	Dec. 31, 2018
Financing Receivable, Past Due [Line Items]
Total Past Due	$ 1,261	$ 3,440
Nonaccrual	17,442	15,210
Current	1,022,468	970,183
Total financing receivables	1,041,171	988,833
Less: Unearned origination fees, net of origination costs	(4,624)	(4,487)
Total loans	1,036,547	984,346
30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	1,261	2,483
Nonaccrual	145	1,329
60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	957
Nonaccrual	1,642	324
Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	8,081	3,597
Commercial
Financing Receivable, Past Due [Line Items]
Total Past Due	237	1,729
Nonaccrual	10,868	12,671
Current	387,126	328,020
Total financing receivables	398,231	342,420
Total loans	396,901	341,091
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	237	872
Nonaccrual	145	1,329
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	857
Nonaccrual	169	324
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	3,857	1,287
Real estate construction one-to-four family
Financing Receivable, Past Due [Line Items]
Total Past Due	30	0
Nonaccrual	1,473	0
Current	35,767	37,111
Total financing receivables	37,270	37,111
Total loans	37,126	36,828
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	30	0
Nonaccrual	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,473
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0
Real estate construction other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	61,446	72,256
Total financing receivables	61,446	72,256
Total loans	60,697	71,658
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	987	1,197
Nonaccrual	3,776	1,694
Current	122,282	123,523
Total financing receivables	127,045	126,414
Total loans	126,381	125,795
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	987	1,197
Nonaccrual	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	2,933	1,694
Real estate term non-owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	332,169	325,720
Total financing receivables	332,169	325,720
Total loans	330,592	324,198
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,198	577
Current	43,944	41,462
Total financing receivables	45,142	42,039
Total loans	44,821	41,746
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,198	577
Consumer secured by 1st deeds of trust
Financing Receivable, Past Due [Line Items]
Total Past Due	0	324
Nonaccrual	0	220
Current	15,952	18,684
Total financing receivables	15,952	19,228
Total loans	15,943	19,234
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	224
Nonaccrual		0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	100
Nonaccrual		0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Consumer other
Financing Receivable, Past Due [Line Items]
Total Past Due	7	190
Nonaccrual	127	48
Current	23,782	23,407
Total financing receivables	23,916	23,645
Total loans	24,086	23,796
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	7	190
Nonaccrual	0	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	$ 93	$ 39