Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
2020	$ 393,833
2019	1,582,645
2018	1,501,216
2017	1,375,311
2016	950,920
Prior	1,950,704
Revolving	997,131
Revolving Converted to Term During the Period	3,109
Total	8,754,869	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	212,035
2019	819,651
2018	850,689
2017	669,837
2016	505,863
Prior	1,033,973
Revolving	13,022
Revolving Converted to Term During the Period	0
Total	4,105,070	4,153,084
Investor loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		4,147,989
Investor loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,151
Investor loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,944
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,040,198	2,070,141
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	104,083
2019	382,006
2018	381,751
2017	302,979
2016	207,185
Prior	644,656
Revolving	11,085
Revolving Converted to Term During the Period	0
Total	2,033,745	2,067,875
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	4,904
Revolving	0
Revolving Converted to Term During the Period	0
Total	4,904	1,178
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	318
2017	0
2016	0
Prior	672
Revolving	559
Revolving Converted to Term During the Period	0
Total	1,549	1,088
Investor loans secured by real estate | Commercial real estate (“CRE”) non-owner-occupied | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,625,682	1,575,726
Investor loans secured by real estate | Multifamily | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	105,208
2019	308,369
2018	315,462
2017	241,938
2016	291,126
Prior	362,377
Revolving	987
Revolving Converted to Term During the Period	0
Total	1,625,467	1,575,510
Investor loans secured by real estate | Multifamily | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0	0
Investor loans secured by real estate | Multifamily | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	215
Revolving	0
Revolving Converted to Term During the Period	0
Total	215	216
Investor loans secured by real estate | Multifamily | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total	377,525	438,786
Investor loans secured by real estate | Construction and land | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	2,250
2019	118,550
2018	139,340
2017	106,230
2016	0
Prior	8,962
Revolving	391
Revolving Converted to Term During the Period	0
Total	375,723	438,769
Investor loans secured by real estate | Construction and land | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0	0
Investor loans secured by real estate | Construction and land | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,802
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,802	17
Investor loans secured by real estate | Construction and land | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	61,665	68,431
Investor loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	494
2019	10,726
2018	12,324
2017	16,189
2016	7,160
Prior	11,032
Revolving	0
Revolving Converted to Term During the Period	0
Total	57,925	65,835
Investor loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	699
2016	0
Prior	271
Revolving	0
Revolving Converted to Term During the Period	0
Total	970	973
Investor loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	1,494
2017	0
2016	392
Prior	884
Revolving	0
Revolving Converted to Term During the Period	0
Total	2,770	1,623
Investor loans secured by real estate | SBA secured by real estate | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	136,092
2019	410,425
2018	398,865
2017	449,016
2016	313,700
Prior	628,168
Revolving	6,434
Revolving Converted to Term During the Period	0
Total	2,342,700	2,288,175
Business loans secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,267,278
Business loans secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		13,930
Business loans secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		6,967
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	83,640	88,381
Business loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	2,109
2019	7,723
2018	14,253
2017	17,388
2016	11,027
Prior	25,896
Revolving	364
Revolving Converted to Term During the Period	0
Total	78,760	83,106
Business loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,015
2016	351
Prior	466
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,832	1,842
Business loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,033
2016	413
Prior	1,602
Revolving	0
Revolving Converted to Term During the Period	0
Total	3,048	3,433
Business loans secured by real estate | SBA secured by real estate | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,887,632	1,846,554
Business loans secured by real estate | CRE owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	114,657
2019	315,128
2018	304,390
2017	311,399
2016	268,077
Prior	540,945
Revolving	5,820
Revolving Converted to Term During the Period	0
Total	1,860,416	1,831,853
Business loans secured by real estate | CRE owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	8,251
2016	0
Prior	6,875
Revolving	0
Revolving Converted to Term During the Period	0
Total	15,126	11,167
Business loans secured by real estate | CRE owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	3,635
2017	727
2016	2,180
Prior	5,298
Revolving	250
Revolving Converted to Term During the Period	0
Total	12,090	3,534
Business loans secured by real estate | CRE owner-occupied | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	371,428	353,240
Business loans secured by real estate | Franchise real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	19,326
2019	87,574
2018	76,587
2017	109,203
2016	31,652
Prior	46,184
Revolving	0
Revolving Converted to Term During the Period	0
Total	370,526	352,319
Business loans secured by real estate | Franchise real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	902
Revolving	0
Revolving Converted to Term During the Period	0
Total	902	921
Business loans secured by real estate | Franchise real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0	0
Business loans secured by real estate | Franchise real estate secured | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	42,151
2019	341,291
2018	233,029
2017	201,482
2016	92,772
Prior	176,139
Revolving	933,054
Revolving Converted to Term During the Period	3,109
Total	2,023,027	1,975,053
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		1,920,189
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		20,641
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		34,223
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,458,969	1,393,270
Commercial loans | Commercial and industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	31,625
2019	125,432
2018	102,018
2017	103,546
2016	37,006
Prior	115,500
Revolving	899,341
Revolving Converted to Term During the Period	1,574
Total	1,416,042	1,359,662
Commercial loans | Commercial and industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	79
2018	352
2017	2,504
2016	137
Prior	1,195
Revolving	18,254
Revolving Converted to Term During the Period	1,250
Total	23,771	13,226
Commercial loans | Commercial and industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	524
2018	2,769
2017	467
2016	1,915
Prior	2,443
Revolving	11,038
Revolving Converted to Term During the Period	0
Total	19,156	20,382
Commercial loans | Commercial and industrial | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	547,793	564,357
Commercial loans | Franchise non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	10,261
2019	212,785
2018	125,954
2017	79,395
2016	52,726
Prior	47,724
Revolving	2,062
Revolving Converted to Term During the Period	0
Total	530,907	546,594
Commercial loans | Franchise non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	3,914
2016	0
Prior	2,861
Revolving	0
Revolving Converted to Term During the Period	0
Total	6,775	6,930
Commercial loans | Franchise non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	9,137
2016	0
Prior	974
Revolving	0
Revolving Converted to Term During the Period	0
Total	10,111	10,833
Commercial loans | Franchise non-real estate secured | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	16,265	17,426
Commercial loans | SBA non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	265
2019	2,383
2018	1,798
2017	2,258
2016	695
Prior	3,737
Revolving	1,572
Revolving Converted to Term During the Period	285
Total	12,993	13,933
Commercial loans | SBA non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	293
Prior	173
Revolving	0
Revolving Converted to Term During the Period	0
Total	466	485
Commercial loans | SBA non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	88
2018	138
2017	261
2016	0
Prior	1,532
Revolving	787
Revolving Converted to Term During the Period	0
Total	2,806	3,008
Commercial loans | SBA non-real estate secured | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	3,555
2019	11,278
2018	18,633
2017	54,976
2016	38,585
Prior	112,424
Revolving	44,621
Revolving Converted to Term During the Period	0
Total	284,072	305,999
Retail loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		305,384
Retail loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Retail loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		615
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total	237,180	255,024
Retail loans | Single family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	3,466
2019	11,064
2018	17,759
2017	16,931
2016	38,556
Prior	108,224
Revolving	39,981
Revolving Converted to Term During the Period	0
Total	235,981	254,463
Retail loans | Single family residential | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	649
Revolving	0
Revolving Converted to Term During the Period	0
Total	649	0
Retail loans | Single family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	192
Revolving	358
Revolving Converted to Term During the Period	0
Total	550	561
Retail loans | Single family residential | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	0
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total	46,892	50,975
Retail loans | Consumer | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	89
2019	214
2018	874
2017	38,045
2016	29
Prior	3,311
Revolving	4,282
Revolving Converted to Term During the Period	0
Total	46,844	50,921
Retail loans | Consumer | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	0	0
Retail loans | Consumer | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	48
Revolving	0
Revolving Converted to Term During the Period	0
Total	48	$ 54
Retail loans | Consumer | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period
Total	$ 0