Loans Receivable and Allowance for Loan Losses - Past Due Financing Receivables (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Financing Receivable, Past Due [Line Items]
Threshold period past due for nonaccrual status of financing receivable	90 days
Past due and Accruing Loans	$ 62,148	$ 56,936
Non-accrual loans	54,785	53,286
Financing Receivable, 90 Days or More Past Due, Still Accruing	7,547	1,529
Guaranteed portion of delinquent SBA loans that are in liquidation	28,100	29,200
Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	41,419	47,005
Non-accrual loans	35,085	42,248
Financing Receivable, 90 Days or More Past Due, Still Accruing	7,547	243
Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	20,729	9,931
Non-accrual loans	19,700	11,038
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	1,286
Real estate-Residential | Legacy
Financing Receivable, Past Due [Line Items]
Non-accrual loans	0	0
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Residential | Acquired
Financing Receivable, Past Due [Line Items]
Non-accrual loans	0	0
Financing Receivable, 90 Days or More Past Due, Still Accruing	0
Real estate-Commercial | Legacy | Retail
Financing Receivable, Past Due [Line Items]
Non-accrual loans	2,029	5,153
Financing Receivable, 90 Days or More Past Due, Still Accruing	449	0
Real estate-Commercial | Legacy | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Non-accrual loans	5,844	7,325
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Legacy | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Non-accrual loans	0	31
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Legacy | Mixed Use
Financing Receivable, Past Due [Line Items]
Non-accrual loans	505	749
Financing Receivable, 90 Days or More Past Due, Still Accruing	634	0
Real estate-Commercial | Legacy | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Non-accrual loans	10,222	6,111
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Legacy | Other
Financing Receivable, Past Due [Line Items]
Non-accrual loans	4,176	5,940
Financing Receivable, 90 Days or More Past Due, Still Accruing	919	0
Real estate-Commercial | Acquired | Retail
Financing Receivable, Past Due [Line Items]
Non-accrual loans	905	829
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Acquired | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Non-accrual loans	5,057	5,500
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	1,286
Real estate-Commercial | Acquired | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Non-accrual loans	271	247
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Acquired | Mixed Use
Financing Receivable, Past Due [Line Items]
Non-accrual loans	160	1,224
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Acquired | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Non-accrual loans	322	349
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Commercial | Acquired | Other
Financing Receivable, Past Due [Line Items]
Non-accrual loans	1,279	259
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Real estate-Construction | Legacy
Financing Receivable, Past Due [Line Items]
Non-accrual loans	0	0
Financing Receivable, 90 Days or More Past Due, Still Accruing	3,850	0
Real estate-Construction | Acquired
Financing Receivable, Past Due [Line Items]
Non-accrual loans	10,165	0
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Commercial Business | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	5,945	12,938
Non-accrual loans	10,204	14,837
Financing Receivable, 90 Days or More Past Due, Still Accruing	1,096	0
Commercial Business | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	525	1,263
Non-accrual loans	689	1,632
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Trade Finance | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	2,697
Non-accrual loans	0	1,661
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Trade Finance | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Non-accrual loans	0	0
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Consumer and Other | Legacy
Financing Receivable, Past Due [Line Items]
Non-accrual loans	2,105	441
Financing Receivable, 90 Days or More Past Due, Still Accruing	599	243
Consumer and Other | Acquired
Financing Receivable, Past Due [Line Items]
Non-accrual loans	852	998
Financing Receivable, 90 Days or More Past Due, Still Accruing	0	0
Consumer | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	11,960	16,800
Consumer | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,458	334
30-59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	15,177	28,467
30-59 Days Past Due | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	12,030	27,137
30-59 Days Past Due | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,147	1,330
30-59 Days Past Due | Commercial Business | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	344	5,500
30-59 Days Past Due | Commercial Business | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	57	397
30-59 Days Past Due | Trade Finance | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,036
30-59 Days Past Due | Trade Finance | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
30-59 Days Past Due | Consumer | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	8,871	16,413
30-59 Days Past Due | Consumer | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	981	0
60-89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	6,494	1,407
60-89 Days Past Due | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	5,384	575
60-89 Days Past Due | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,110	832
60-89 Days Past Due | Commercial Business | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	126	435
60-89 Days Past Due | Commercial Business | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	226	613
60-89 Days Past Due | Trade Finance | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
60-89 Days Past Due | Trade Finance | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
60-89 Days Past Due | Consumer | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	914	140
60-89 Days Past Due | Consumer | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	40,477	27,062
90 or More Days Past Due | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	24,005	19,293
90 or More Days Past Due | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	16,472	7,769
90 or More Days Past Due | Commercial Business | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	5,475	7,003
90 or More Days Past Due | Commercial Business | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	242	253
90 or More Days Past Due | Trade Finance | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,661
90 or More Days Past Due | Trade Finance | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
90 or More Days Past Due | Consumer | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	2,175	247
90 or More Days Past Due | Consumer | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	477	334
Real estate-Construction | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,850	0
Real estate-Construction | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	10,165	0
Real estate-Construction | 30-59 Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Construction | 30-59 Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Construction | 60-89 Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Construction | 60-89 Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Construction | 90 or More Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,850	0
Real estate-Construction | 90 or More Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	10,165	0
Real estate-Commercial | Real Estate | Legacy | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	4,924	1,542
Real estate-Commercial | Real Estate | Legacy | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,782	5,368
Real estate-Commercial | Real Estate | Legacy | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	2,302	31
Real estate-Commercial | Real Estate | Legacy | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,394	0
Real estate-Commercial | Real Estate | Legacy | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,853	3,387
Real estate-Commercial | Real Estate | Legacy | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,409	4,242
Real estate-Commercial | Real Estate | Acquired | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	620	949
Real estate-Commercial | Real Estate | Acquired | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	4,909	5,206
Real estate-Commercial | Real Estate | Acquired | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	929	375
Real estate-Commercial | Real Estate | Acquired | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,141
Real estate-Commercial | Real Estate | Acquired | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	232	261
Real estate-Commercial | Real Estate | Acquired | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,891	402
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,083	733
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	821	153
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	318	0
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	593	0
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,465
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Legacy | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,837
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	347
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	525	0
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	679	154
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	107
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	94	142
Real estate-Commercial | 30-59 Days Past Due | Real Estate | Acquired | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	811	183
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,424	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	936	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	1,984	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Legacy | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	54	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	45	0
Real estate-Commercial | 60-89 Days Past Due | Real Estate | Acquired | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	785	219
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	2,417	809
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	2,025	5,215
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	31
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	801	0
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,853	1,922
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Legacy | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	3,409	2,405
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Retail
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	620	602
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Hotel & Motel
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	4,384	5,206
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Gas Station & Car Wash
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	196	221
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Mixed Use
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	1,034
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Industrial & Warehouse
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	93	119
Real estate-Commercial | 90 or More Days Past Due | Real Estate | Acquired | Other
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	295	0
Real estate-Residential | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 30-59 Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 30-59 Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 60-89 Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 60-89 Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 90 or More Days Past Due | Real Estate | Legacy
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	0	0
Real estate-Residential | 90 or More Days Past Due | Real Estate | Acquired
Financing Receivable, Past Due [Line Items]
Past due and Accruing Loans	$ 0	$ 0