Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	$ 16,395	$ 15,488
Charge-offs	(525)	(310)
Recoveries	6	6
Net (charge-offs) recoveries	(519)	(304)
Provision for loan losses charged to expense	1,500	500
Ending Balance	17,376	15,684
SBA loans held for investment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,079	1,655
Charge-offs	(25)	(308)
Recoveries	5	1
Net (charge-offs) recoveries	(20)	(307)
Provision for loan losses charged to expense	(54)	330
Ending Balance	1,005	1,678
Total commercial loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	9,722	8,705
Charge-offs	(300)	(1)
Recoveries	1	5
Net (charge-offs) recoveries	(299)	4
Provision for loan losses charged to expense	706	84
Ending Balance	10,129	8,793
Residential mortgage loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	4,254	3,900
Charge-offs	(200)	0
Recoveries	0	0
Net (charge-offs) recoveries	(200)	0
Provision for loan losses charged to expense	709	116
Ending Balance	4,763	4,016
Total consumer loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,340	1,228
Charge-offs	0	(1)
Recoveries	0	0
Net (charge-offs) recoveries	0	(1)
Provision for loan losses charged to expense	139	(30)
Ending Balance	$ 1,479	$ 1,197