Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Past Due [Line Items]
Total	$ 15,082,884	$ 8,722,311
Current
Financing Receivable, Past Due [Line Items]
Current	15,044,696	8,703,209
30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	6,248	2,104
60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	4,133	10,559
90
Financing Receivable, Past Due [Line Items]
Days Past Due	27,807	6,439
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	8,426,157	4,153,084
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,783,692	2,070,141
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Total	5,225,557	1,575,726
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Total	357,426	438,786
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	59,482	68,431
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	8,416,282	4,150,427
Investor loans secured by real estate | Current | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,780,620	2,067,874
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Current	5,222,439	1,575,726
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Current	354,729	438,786
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	58,494	68,041
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	4,013	1,179
Investor loans secured by real estate | 30-59 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1,179
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	3,118	0
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	895	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	5,862	1,478
Investor loans secured by real estate | 90 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	3,072	1,088
Investor loans secured by real estate | 90 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	1,802	0
Investor loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	988	390
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	2,620,343	2,288,175
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	85,542	88,381
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,170,154	1,846,554
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Total	364,647	353,240
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	2,612,025	2,286,409
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	84,536	86,946
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,162,842	1,846,223
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Current	364,647	353,240
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	1,062	331
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	1,062	331
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	319	589
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	589
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	319	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	6,937	846
Business loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	1,006	846
Business loans secured by real estate | 90 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	5,931	0
Business loans secured by real estate | 90 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Total	3,724,905	1,975,053
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Total	2,051,313	1,393,270
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	523,755	564,357
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	21,057	17,426
Commercial loans | SBA PPP
Financing Receivable, Past Due [Line Items]
Total	1,128,780	0
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Current	3,705,657	1,960,382
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Current	2,041,675	1,389,026
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	515,313	555,215
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	19,889	16,141
Commercial loans | Current | SBA PPP
Financing Receivable, Past Due [Line Items]
Current	1,128,780
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	796	589
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	796	422
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	167
Commercial loans | 30-59 | SBA PPP
Financing Receivable, Past Due [Line Items]
Days Past Due	0
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	3,444	9,968
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	3,116	826
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	9,142
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	328	0
Commercial loans | 60-89 | SBA PPP
Financing Receivable, Past Due [Line Items]
Days Past Due	0
Commercial loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	15,008	4,114
Commercial loans | 90 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	5,726	2,996
Commercial loans | 90 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	8,442	0
Commercial loans | 90 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	840	1,118
Commercial loans | 90 | SBA PPP
Financing Receivable, Past Due [Line Items]
Days Past Due	0
Retail loans
Financing Receivable, Past Due [Line Items]
Total	311,479	305,999
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Total	265,170	255,024
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Total	46,309	50,975
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Current	310,732	305,991
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Current	264,549	255,024
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Current	46,183	50,967
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	377	5
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	257	0
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	120	5
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	370	2
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	364	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	6	2
Retail loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1
Retail loans | 90 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	$ 0	$ 1