Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2019	Mar. 31, 2018
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	$ 15,488	$ 13,556
Charge-offs	(310)	(87)
Recoveries	6	227
Net (charge-offs) recoveries	(304)	140
Provision for loan losses charged to expense	500	500
Ending Balance	15,684	14,196
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,655	1,471
Charge-offs	(308)	(81)
Recoveries	1	64
Net (charge-offs) recoveries	(307)	(17)
Provision for loan losses charged to expense	330	50
Ending Balance	1,678	1,504
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	8,705	7,825
Charge-offs	(1)	0
Recoveries	5	16
Net (charge-offs) recoveries	4	16
Provision for loan losses charged to expense	84	298
Ending Balance	8,793	8,139
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	3,900	3,130
Charge-offs	0	0
Recoveries	0	13
Net (charge-offs) recoveries	0	13
Provision for loan losses charged to expense	116	287
Ending Balance	4,016	3,430
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,228	1,130
Charge-offs	(1)	(6)
Recoveries	0	134
Net (charge-offs) recoveries	(1)	128
Provision for loan losses charged to expense	(30)	(135)
Ending Balance	$ 1,197	$ 1,123