Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Basis of Impairment Method (Details) - USD ($) $ in Thousands	Mar. 31, 2019	Dec. 31, 2018	Mar. 31, 2018	Dec. 31, 2017
Financing Receivable, Allowance for Credit Losses [Line Items]
Total	$ 15,684	$ 15,488	$ 14,196	$ 13,556
Total loans	1,304,984	1,293,395
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	140	540
Collectively evaluated for impairment	1,538	1,115
Total	1,678	1,655	1,504	1,471
Individually evaluated for impairment	746	1,471
Collectively evaluated for impairment	38,069	37,862
Total loans	38,815	39,333
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	90	127
Collectively evaluated for impairment	8,703	8,578
Total	8,793	8,705	8,139	7,825
Individually evaluated for impairment	1,784	1,821
Collectively evaluated for impairment	700,451	692,281
Total loans	702,235	694,102
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	4,016	3,900
Total	4,016	3,900	3,430	3,130
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	438,431	436,056
Total loans	438,431	436,056
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	1,197	1,228
Total	1,197	1,228	$ 1,123	$ 1,130
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	125,503	123,904
Total loans	125,503	123,904
Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	230	667
Collectively evaluated for impairment	15,454	14,821
Total	15,684	15,488
Individually evaluated for impairment	2,530	3,292
Collectively evaluated for impairment	1,302,454	1,290,103
Total loans	$ 1,304,984	$ 1,293,395