Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
2020	$ 2,183,956
2019	3,628,848
2018	2,540,813
2017	1,998,126
2016	1,281,491
Prior	2,541,780
Revolving	901,170
Revolving Converted to Term During the Period	6,700
Total	15,082,884	$ 8,722,311
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		8,640,840
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		36,722
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		44,749
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	692,143
2019	2,473,310
2018	1,661,597
2017	1,175,829
2016	802,350
Prior	1,607,953
Revolving	12,975
Revolving Converted to Term During the Period	0
Total	8,426,157	4,153,084
Investor loans secured by real estate \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		4,147,989
Investor loans secured by real estate \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,151
Investor loans secured by real estate \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,944
Investor loans secured by real estate \| CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,783,692	2,070,141
Investor loans secured by real estate \| CRE non-owner-occupied \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	155,429
2019	567,989
2018	482,567
2017	359,173
2016	319,468
Prior	876,776
Revolving	10,924
Revolving Converted to Term During the Period	0
Total	2,772,326	2,067,875
Investor loans secured by real estate \| CRE non-owner-occupied \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	4,872
Revolving	0
Revolving Converted to Term During the Period	0
Total	4,872	1,178
Investor loans secured by real estate \| CRE non-owner-occupied \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	322
2017	528
2016	518
Prior	4,567
Revolving	559
Revolving Converted to Term During the Period	0
Total	6,494	1,088
Investor loans secured by real estate \| Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total	5,225,557	1,575,726
Investor loans secured by real estate \| Multifamily \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	522,381
2019	1,753,769
2018	1,010,047
2017	761,576
2016	474,835
Prior	701,092
Revolving	1,031
Revolving Converted to Term During the Period	0
Total	5,224,731	1,575,510
Investor loans secured by real estate \| Multifamily \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Investor loans secured by real estate \| Multifamily \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	826
Revolving	0
Revolving Converted to Term During the Period	0
Total	826	216
Investor loans secured by real estate \| Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total	357,426	438,786
Investor loans secured by real estate \| Construction and land \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	13,838
2019	140,848
2018	135,797
2017	36,569
2016	0
Prior	8,468
Revolving	461
Revolving Converted to Term During the Period	0
Total	335,981	438,769
Investor loans secured by real estate \| Construction and land \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	19,643
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	19,643	0
Investor loans secured by real estate \| Construction and land \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,802
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,802	17
Investor loans secured by real estate \| SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	59,482	68,431
Investor loans secured by real estate \| SBA secured by real estate \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	495
2019	10,436
2018	12,284
2017	15,483
2016	6,734
Prior	9,392
Revolving	0
Revolving Converted to Term During the Period	0
Total	54,824	65,835
Investor loans secured by real estate \| SBA secured by real estate \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	698
2016	0
Prior	269
Revolving	0
Revolving Converted to Term During the Period	0
Total	967	973
Investor loans secured by real estate \| SBA secured by real estate \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	268
2018	937
2017	0
2016	795
Prior	1,691
Revolving	0
Revolving Converted to Term During the Period	0
Total	3,691	1,623
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
2020	240,336
2019	548,159
2018	443,266
2017	453,579
2016	310,433
Prior	619,080
Revolving	5,490
Revolving Converted to Term During the Period	0
Total	2,620,343	2,288,175
Business loans secured by real estate \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		2,267,278
Business loans secured by real estate \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		13,930
Business loans secured by real estate \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		6,967
Business loans secured by real estate \| SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	85,542	88,381
Business loans secured by real estate \| SBA secured by real estate \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,905
2019	7,693
2018	14,159
2017	17,302
2016	10,510
Prior	28,021
Revolving	365
Revolving Converted to Term During the Period	0
Total	79,955	83,106
Business loans secured by real estate \| SBA secured by real estate \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,008
2016	343
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,351	1,842
Business loans secured by real estate \| SBA secured by real estate \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	914
2016	148
Prior	3,174
Revolving	0
Revolving Converted to Term During the Period	0
Total	4,236	3,433
Business loans secured by real estate \| CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,170,154	1,846,554
Business loans secured by real estate \| CRE owner-occupied \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	212,046
2019	435,924
2018	345,829
2017	324,096
2016	261,057
Prior	525,237
Revolving	4,875
Revolving Converted to Term During the Period	0
Total	2,109,064	1,831,853
Business loans secured by real estate \| CRE owner-occupied \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	5,979
2019	15,734
2018	0
2017	3,886
2016	0
Prior	8,476
Revolving	0
Revolving Converted to Term During the Period	0
Total	34,075	11,167
Business loans secured by real estate \| CRE owner-occupied \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	7,086
2017	2,525
2016	6,797
Prior	10,357
Revolving	250
Revolving Converted to Term During the Period	0
Total	27,015	3,534
Business loans secured by real estate \| Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	364,647	353,240
Business loans secured by real estate \| Franchise real estate secured \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	20,406
2019	88,808
2018	76,192
2017	103,848
2016	31,578
Prior	42,932
Revolving	0
Revolving Converted to Term During the Period	0
Total	363,764	352,319
Business loans secured by real estate \| Franchise real estate secured \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		921
Business loans secured by real estate \| Franchise real estate secured \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	883
Revolving	0
Revolving Converted to Term During the Period	0
Total	883	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,246,542
2019	597,779
2018	419,527
2017	315,688
2016	130,311
Prior	166,605
Revolving	841,753
Revolving Converted to Term During the Period	6,700
Total	3,724,905	1,975,053
Commercial loans \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		1,920,189
Commercial loans \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		20,641
Commercial loans \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		34,223
Commercial loans \| Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,051,313	1,393,270
Commercial loans \| Commercial and industrial \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	106,699
2019	388,548
2018	290,535
2017	221,262
2016	71,937
Prior	105,707
Revolving	803,473
Revolving Converted to Term During the Period	2,470
Total	1,990,631	1,359,662
Commercial loans \| Commercial and industrial \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	73
2018	2,928
2017	1,994
2016	1,562
Prior	2,671
Revolving	20,969
Revolving Converted to Term During the Period	500
Total	30,697	13,226
Commercial loans \| Commercial and industrial \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	70
2019	309
2018	2,621
2017	1,000
2016	6,567
Prior	4,883
Revolving	14,342
Revolving Converted to Term During the Period	193
Total	29,985	20,382
Commercial loans \| Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	523,755	564,357
Commercial loans \| Franchise non-real estate secured \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	10,347
2019	206,422
2018	121,616
2017	74,743
2016	49,307
Prior	45,116
Revolving	2,205
Revolving Converted to Term During the Period	0
Total	509,756	546,594
Commercial loans \| Franchise non-real estate secured \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	3,834
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	3,834	6,930
Commercial loans \| Franchise non-real estate secured \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	7,737
2016	0
Prior	2,428
Revolving	0
Revolving Converted to Term During the Period	0
Total	10,165	10,833
Commercial loans \| SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	21,057	17,426
Commercial loans \| SBA non-real estate secured \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	646
2019	2,341
2018	1,428
2017	2,517
2016	654
Prior	4,276
Revolving	0
Revolving Converted to Term During the Period	3,537
Total	15,399	13,933
Commercial loans \| SBA non-real estate secured \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	1,745
2016	284
Prior	150
Revolving	0
Revolving Converted to Term During the Period	0
Total	2,179	485
Commercial loans \| SBA non-real estate secured \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	86
2018	399
2017	856
2016	0
Prior	1,374
Revolving	764
Revolving Converted to Term During the Period	0
Total	3,479	3,008
Commercial loans \| SBA PPP
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,128,780	0
Commercial loans \| SBA PPP \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,128,780
2019	0
2018	0
2017	0
2016	0
Prior	0
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,128,780
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	4,935
2019	9,600
2018	16,423
2017	53,030
2016	38,397
Prior	148,142
Revolving	40,952
Revolving Converted to Term During the Period	0
Total	311,479	305,999
Retail loans \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total		305,384
Retail loans \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total		0
Retail loans \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total		615
Retail loans \| Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total	265,170	255,024
Retail loans \| Single family residential \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	4,863
2019	9,426
2018	15,553
2017	14,991
2016	38,123
Prior	143,690
Revolving	37,140
Revolving Converted to Term During the Period	0
Total	263,786	254,463
Retail loans \| Single family residential \| Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	0
Prior	58
Revolving	0
Revolving Converted to Term During the Period	0
Total	58	0
Retail loans \| Single family residential \| Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0
2019	0
2018	0
2017	0
2016	251
Prior	1,075
Revolving	0
Revolving Converted to Term During the Period	0
Total	1,326	561
Retail loans \| Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total	46,309	50,975
Retail loans \| Consumer \| Pass
Financing Receivable, Credit Quality Indicator [Line Items]
2020	72
2019	174

Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($)
$ in Thousands

Jun. 30, 2020

Dec. 31, 2019

Financing Receivable, Credit Quality Indicator [Line Items]

$ 2,183,956

3,628,848

2,540,813

1,998,126