Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Past Due [Line Items]
Total Past Due	$ 861	$ 1,534
Nonaccrual	14,365	15,356
Current	1,432,982	1,031,566
Total financing receivable	1,448,208	1,048,456
Less: Unearned origination fees, net of origination costs	(15,007)	(5,085)
Total loans	1,433,201	1,043,371
30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	250	1,534
Nonaccrual	354	1,937
60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	611	0
Nonaccrual	60	474
Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	10,154	6,010
Commercial
Financing Receivable, Past Due [Line Items]
Total Past Due	250	270
Nonaccrual	8,362	9,153
Current	771,548	403,267
Total financing receivable	780,160	412,690
Total loans	768,988	411,327
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	250	270
Nonaccrual	354	270
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	385
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	6,031	2,862
Real estate construction one-to-four family
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	702	1,349
Current	37,616	37,469
Total financing receivable	38,318	38,818
Total loans	38,148	38,503
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	702	1,349
Real estate construction other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	76,146	61,808
Total financing receivable	76,146	61,808
Total loans	75,333	60,906
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	0	338
Nonaccrual	3,947	3,489
Current	150,794	135,064
Total financing receivable	154,741	138,891
Total loans	153,942	138,181
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	338
Nonaccrual	0	1,641
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	60	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	2,245	623
Real estate term non-owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	319,257	312,960
Total financing receivable	319,257	312,960
Total loans	317,333	311,302
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	26
Nonaccrual	1,176	1,176
Current	40,100	41,304
Total financing receivable	41,276	42,506
Total loans	40,989	42,200
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	26
Nonaccrual	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,176	1,176
Consumer secured by 1st deed of trust
Financing Receivable, Past Due [Line Items]
Total Past Due	553	750
Nonaccrual	66	68
Current	13,891	15,380
Total financing receivable	14,510	16,198
Total loans	14,496	16,191
Consumer secured by 1st deed of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	750
Nonaccrual	0	0
Consumer secured by 1st deed of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	553	0
Nonaccrual	0	0
Consumer secured by 1st deed of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer other
Financing Receivable, Past Due [Line Items]
Total Past Due	58	150
Nonaccrual	112	121
Current	23,630	24,314
Total financing receivable	23,800	24,585
Total loans	23,972	24,761
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	150
Nonaccrual	0	26
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	58	0
Nonaccrual	0	89
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	$ 0	$ 0