Loans and Allowance for Loan Losses - Geographic and Collateral Concentration (Details)	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Geographical concentration
Concentration risk
Percentage of loan	100.00%	100.00%
Geographical concentration | California
Concentration risk
Percentage of loan	18.10%	16.90%
Geographical concentration | Texas
Concentration risk
Percentage of loan	14.20%	15.20%
Geographical concentration | New York
Concentration risk
Percentage of loan	9.80%	8.30%
Geographical concentration | Florida
Concentration risk
Percentage of loan	7.80%	8.30%
Geographical concentration | Illinois
Concentration risk
Percentage of loan	5.20%	5.20%
Geographical concentration | Georgia
Concentration risk
Percentage of loan	4.90%	4.80%
Geographical concentration | North Carolina
Concentration risk
Percentage of loan	3.10%	3.20%
Geographical concentration | Washington
Concentration risk
Percentage of loan	3.10%	2.80%
Geographical concentration | Arizona
Concentration risk
Percentage of loan	2.80%	3.40%
Geographical concentration | Colorado
Concentration risk
Percentage of loan	2.80%	2.80%
Geographical concentration | Other
Concentration risk
Percentage of loan	28.20%	29.10%
Collateral concentration
Concentration risk
Percentage of loan	100.00%	100.00%
Percentage of SBA loan	100.00%	100.00%
Collateral concentration | Lodging
Concentration risk
Percentage of SBA loan	17.20%	17.30%
Collateral concentration | Offices of Physicians
Concentration risk
Percentage of SBA loan	12.00%	14.10%
Collateral concentration | Child Day Care Services
Concentration risk
Percentage of SBA loan	7.20%	8.10%
Collateral concentration | Eating Places
Concentration risk
Percentage of SBA loan	5.30%	6.10%
Collateral concentration | Gasoline Service Stations
Concentration risk
Percentage of SBA loan	3.40%	3.70%
Collateral concentration | Veterinarians
Concentration risk
Percentage of SBA loan	3.30%	4.10%
Collateral concentration | Executive Search Services
Concentration risk
Percentage of SBA loan	2.60%
Collateral concentration | Other Miscellaneous Store Retailers Except Tobacco Stores
Concentration risk
Percentage of SBA loan	2.30%
Collateral concentration | Funeral Service and Crematories
Concentration risk
Percentage of SBA loan	1.80%	2.00%
Collateral concentration | Grocery Stores
Concentration risk
Percentage of SBA loan	1.70%	2.00%
Collateral concentration | Other
Concentration risk
Percentage of SBA loan	43.20%	42.60%
Collateral concentration | Multi-family
Concentration risk
Percentage of loan	23.80%	26.60%
Collateral concentration | SBA
Concentration risk
Percentage of loan	17.40%	17.60%
Collateral concentration | Retail
Concentration risk
Percentage of loan	17.30%	17.50%
Collateral concentration | Office
Concentration risk
Percentage of loan	13.10%	12.90%
Collateral concentration | Mixed Use
Concentration risk
Percentage of loan	13.10%	10.40%
Collateral concentration | Industrial
Concentration risk
Percentage of loan	7.00%	6.40%
Collateral concentration | Lodging/Residential
Concentration risk
Percentage of loan	3.20%	3.30%
Collateral concentration | Other
Concentration risk
Percentage of loan	5.10%	5.30%