Loans and Allowance for Loan Losses - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	$ 1,578,804	$ 1,450,831
COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	171,624
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	1,569,198	1,440,186
Pass | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	167,583
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	4,746	1,783
Special Mention | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	4,041
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	4,860	8,862
Substandard | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Doubtful | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	1,214,353	1,187,061
Real Estate | Commercial property
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	853,708	803,014
Real Estate | Commercial property | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	137,562
Real Estate | Commercial property | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	852,325	802,373
Real Estate | Commercial property | Pass | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	137,219
Real Estate | Commercial property | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	1,048	0
Real Estate | Commercial property | Special Mention | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	343
Real Estate | Commercial property | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	335	641
Real Estate | Commercial property | Substandard | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate | Commercial property | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Commercial property | Doubtful | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate | Residential property
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	212,804	235,046
Real Estate | Residential property | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	19,233
Real Estate | Residential property | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	212,804	235,046
Real Estate | Residential property | Pass | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	19,233
Real Estate | Residential property | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Residential property | Special Mention | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate | Residential property | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Residential property | Substandard | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate | Residential property | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Residential property | Doubtful | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Real Estate | SBA property
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	128,038	129,837
Real Estate | SBA property | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	125,497	124,135
Real Estate | SBA property | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	72
Real Estate | SBA property | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	2,541	5,630
Real Estate | SBA property | Substandard | Loans guaranteed by US government agency
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	114	2,400
Real Estate | SBA property | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Construction
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	19,803	19,164
Real Estate | Construction | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	19,803	17,453
Real Estate | Construction | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	1,711
Real Estate | Construction | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Real Estate | Construction | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	342,518	240,480
Commercial and Industrial | Commercial term
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	90,867	103,380
Commercial and Industrial | Commercial term | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	14,757
Commercial and Industrial | Commercial term | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	87,241	103,380
Commercial and Industrial | Commercial term | Pass | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	11,131
Commercial and Industrial | Commercial term | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	3,626	0
Commercial and Industrial | Commercial term | Special Mention | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	3,626
Commercial and Industrial | Commercial term | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial | Commercial term | Substandard | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | Commercial term | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial | Commercial term | Doubtful | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | Commercial lines of credit
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	92,222	111,768
Commercial and Industrial | Commercial lines of credit | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	90,697	109,880
Commercial and Industrial | Commercial lines of credit | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial | Commercial lines of credit | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	1,525	1,888
Commercial and Industrial | Commercial lines of credit | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial | SBA commercial term
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	23,011	25,332
Commercial and Industrial | SBA commercial term | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	72
Commercial and Industrial | SBA commercial term | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	22,547	24,677
Commercial and Industrial | SBA commercial term | Pass | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | SBA commercial term | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	72	0
Commercial and Industrial | SBA commercial term | Special Mention | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	72
Commercial and Industrial | SBA commercial term | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	392	655
Commercial and Industrial | SBA commercial term | Substandard | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | SBA commercial term | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Commercial and Industrial | SBA commercial term | Doubtful | COVID-19 Loan Modification
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | SBA PPP
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	136,418	0
Commercial and Industrial | SBA PPP | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	136,418
Commercial and Industrial | SBA PPP | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | SBA PPP | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Commercial and Industrial | SBA PPP | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0
Other consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	21,933	23,290
Other consumer loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	21,866	23,242
Other consumer loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	0	0
Other consumer loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	67	48
Other consumer loans | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held-for-investment	$ 0	$ 0