Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Past Due [Line Items]
Total	$ 8,754,869	$ 8,722,311
Current
Financing Receivable, Past Due [Line Items]
Current	8,725,998	8,703,209
30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	8,285	2,104
60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	1,502	10,559
90
Financing Receivable, Past Due [Line Items]
Days Past Due	19,084	6,439
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	4,105,070	4,153,084
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,040,198	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Total	1,625,682	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Total	377,525	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	61,665	68,431
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	4,097,513	4,150,427
Investor loans secured by real estate | Current | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,037,130	2,067,874
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Current	1,625,682	1,575,726
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Current	375,723	438,786
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	58,978	68,041
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	3,338	1,179
Investor loans secured by real estate | 30-59 | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	2,191	1,179
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	1,147	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	1,148	0
Investor loans secured by real estate | 60-89 | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	1,148	0
Investor loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	3,071	1,478
Investor loans secured by real estate | 90 | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	877	1,088
Investor loans secured by real estate | 90 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	1,802	0
Investor loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	392	390
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	2,342,700	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	83,640	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Total	1,887,632	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Total	371,428	353,240
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	2,338,032	2,286,409
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	82,608	86,946
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Current	1,883,996	1,846,223
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Current	371,428	353,240
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	3,636	331
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	3,636	331
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	589
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	589
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	1,032	846
Business loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	1,032	846
Business loans secured by real estate | 90 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 90 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Total	2,023,027	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Total	1,458,969	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	547,793	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	16,265	17,426
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Current	2,006,381	1,960,382
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Current	1,452,405	1,389,026
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	538,651	555,215
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	15,325	16,141
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	1,311	589
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	1,249	422
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	62	167
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	354	9,968
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	354	826
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	9,142
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	14,981	4,114
Commercial loans | 90 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	4,961	2,996
Commercial loans | 90 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	9,142	0
Commercial loans | 90 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	878	1,118
Retail loans
Financing Receivable, Past Due [Line Items]
Total	284,072	305,999
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Total	237,180	255,024
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Total	46,892	50,975
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Current	284,072	305,991
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Current	237,180	255,024
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Current	46,892	50,967
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	0	5
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	5
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	2
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	2
Retail loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1
Retail loans | 90 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	$ 0	$ 1