Loans Held for Investment - Internal Risk Grading System (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	$ 8,754,869	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	4,105,070	4,153,084
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,040,198	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,625,682	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	377,525	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	61,665	68,431
Investor loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		4,147,989
Investor loans secured by real estate | Pass | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,033,745	2,067,875
Investor loans secured by real estate | Pass | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,625,467	1,575,510
Investor loans secured by real estate | Pass | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	375,723	438,769
Investor loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	57,925	65,835
Investor loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,151
Investor loans secured by real estate | Special Mention | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	4,904	1,178
Investor loans secured by real estate | Special Mention | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	0	0
Investor loans secured by real estate | Special Mention | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	0	0
Investor loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	970	973
Investor loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,944
Investor loans secured by real estate | Substandard | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,549	1,088
Investor loans secured by real estate | Substandard | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	215	216
Investor loans secured by real estate | Substandard | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,802	17
Investor loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,770	1,623
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,342,700	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	83,640	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,887,632	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	371,428	353,240
Business loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		2,267,278
Business loans secured by real estate | Pass | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	78,760	83,106
Business loans secured by real estate | Pass | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,860,416	1,831,853
Business loans secured by real estate | Pass | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	370,526	352,319
Business loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		13,930
Business loans secured by real estate | Special Mention | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,832	1,842
Business loans secured by real estate | Special Mention | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	15,126	11,167
Business loans secured by real estate | Special Mention | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	902	921
Business loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		6,967
Business loans secured by real estate | Substandard | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	3,048	3,433
Business loans secured by real estate | Substandard | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	12,090	3,534
Business loans secured by real estate | Substandard | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	0	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,023,027	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,458,969	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	547,793	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	16,265	17,426
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		1,920,189
Commercial loans | Pass | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	1,416,042	1,359,662
Commercial loans | Pass | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	530,907	546,594
Commercial loans | Pass | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	12,993	13,933
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		20,641
Commercial loans | Special Mention | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	23,771	13,226
Commercial loans | Special Mention | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	6,775	6,930
Commercial loans | Special Mention | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	466	485
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		34,223
Commercial loans | Substandard | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	19,156	20,382
Commercial loans | Substandard | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	10,111	10,833
Commercial loans | Substandard | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	2,806	3,008
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	284,072	305,999
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	237,180	255,024
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	46,892	50,975
Retail loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		305,384
Retail loans | Pass | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	235,981	254,463
Retail loans | Pass | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	46,844	50,921
Retail loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		0
Retail loans | Special Mention | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	649	0
Retail loans | Special Mention | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	0	0
Retail loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans		615
Retail loans | Substandard | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	550	561
Retail loans | Substandard | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total gross loans	$ 48	$ 54