Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	$ 1,448,208	$ 1,048,456
Less: Unearned origination fees, net of origination costs	(15,007)	(5,085)
Total loans	1,433,201	1,043,371
AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,401,074	994,583
AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	25,923	29,660
AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	21,099	24,213
AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	46
AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	66
Commercial
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	780,160	412,690
Total loans	768,988	411,327
Commercial | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	766,043	394,107
Commercial | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	2,465	2,279
Commercial | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	11,606	16,304
Commercial | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	46
Commercial | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction one-to-four family
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	38,318	38,818
Total loans	38,148	38,503
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	36,329	34,132
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,287	3,337
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	702	1,349
Real estate construction one-to-four family | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction one-to-four family | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	76,146	61,808
Total loans	75,333	60,906
Real estate construction other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	76,146	61,808
Real estate construction other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Real estate construction other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term owner occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	154,741	138,891
Total loans	153,942	138,181
Real estate term owner occupied | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	144,310	129,959
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	3,637	3,828
Real estate term owner occupied | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	6,794	5,104
Real estate term owner occupied | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term owner occupied | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term non-owner occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	319,257	312,960
Total loans	317,333	311,302
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	301,454	295,482
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	17,178	17,478
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	625	0
Real estate term non-owner occupied | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term non-owner occupied | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	41,276	42,506
Total loans	40,989	42,200
Real estate term other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	38,921	38,771
Real estate term other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,179	2,559
Real estate term other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,176	1,176
Real estate term other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer secured by 1st deed of trust
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	14,510	16,198
Total loans	14,496	16,191
Consumer secured by 1st deed of trust | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	14,183	15,860
Consumer secured by 1st deed of trust | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	177	179
Consumer secured by 1st deed of trust | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	84	159
Consumer secured by 1st deed of trust | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer secured by 1st deed of trust | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	66
Consumer other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	23,800	24,585
Total loans	23,972	24,761
Consumer other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	23,688	24,464
Consumer other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Consumer other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	112	$ 121
Consumer other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	$ 0