Loans Held for Investment - Delinquencies (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Financing Receivable, Past Due [Line Items]
Total	$ 13,117,392	$ 13,236,433
Current
Financing Receivable, Past Due [Line Items]
Current	13,094,805	13,223,111
30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	13,116	1,269
60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	61	57
90
Financing Receivable, Past Due [Line Items]
Days Past Due	9,410	11,996
Investor loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	8,412,216	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,729,785	2,675,085
Investor loans secured by real estate | Multifamily
Financing Receivable, Past Due [Line Items]
Total	5,309,592	5,171,356
Investor loans secured by real estate | Construction and land
Financing Receivable, Past Due [Line Items]
Total	316,458	321,993
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	56,381	57,331
Investor loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	8,401,485	8,223,750
Investor loans secured by real estate | Current | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,719,494	2,674,328
Investor loans secured by real estate | Current | Multifamily
Financing Receivable, Past Due [Line Items]
Current	5,309,592	5,171,355
Investor loans secured by real estate | Current | Construction and land
Financing Receivable, Past Due [Line Items]
Current	316,458	321,993
Investor loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	55,941	56,074
Investor loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	9,743	1
Investor loans secured by real estate | 30-59 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	9,743	0
Investor loans secured by real estate | 30-59 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1
Investor loans secured by real estate | 30-59 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	988	2,014
Investor loans secured by real estate | 90 | CRE non-owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	548	757
Investor loans secured by real estate | 90 | Multifamily
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90 | Construction and land
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Investor loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	440	1,257
Business loans secured by real estate
Financing Receivable, Past Due [Line Items]
Total	2,444,756	2,541,577
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Total	73,967	79,595
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Total	2,029,984	2,114,050
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Total	340,805	347,932
Business loans secured by real estate | Current
Financing Receivable, Past Due [Line Items]
Current	2,438,792	2,534,714
Business loans secured by real estate | Current | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Current	73,307	78,036
Business loans secured by real estate | Current | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Current	2,024,680	2,108,746
Business loans secured by real estate | Current | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Current	340,805	347,932
Business loans secured by real estate | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	0	486
Business loans secured by real estate | 30-59 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	486
Business loans secured by real estate | 30-59 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 30-59 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 60-89 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Business loans secured by real estate | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	5,964	6,377
Business loans secured by real estate | 90 | SBA secured by real estate
Financing Receivable, Past Due [Line Items]
Days Past Due	660	1,073
Business loans secured by real estate | 90 | CRE owner-occupied
Financing Receivable, Past Due [Line Items]
Days Past Due	5,304	5,304
Business loans secured by real estate | 90 | Franchise real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans
Financing Receivable, Past Due [Line Items]
Total	2,070,047	2,229,588
Commercial loans | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Total	1,656,098	1,768,834
Commercial loans | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	399,041	444,797
Commercial loans | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Total	14,908	15,957
Commercial loans | Current
Financing Receivable, Past Due [Line Items]
Current	2,064,155	2,225,160
Commercial loans | Current | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Current	1,651,203	1,765,451
Commercial loans | Current | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	399,041	444,797
Commercial loans | Current | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Current	13,911	14,912
Commercial loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	3,373	766
Commercial loans | 30-59 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	3,068	428
Commercial loans | 30-59 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 30-59 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	305	338
Commercial loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	61	57
Commercial loans | 60-89 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	61	57
Commercial loans | 60-89 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 60-89 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	2,458	3,605
Commercial loans | 90 | Commercial and industrial
Financing Receivable, Past Due [Line Items]
Days Past Due	1,766	2,898
Commercial loans | 90 | Franchise non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Commercial loans | 90 | SBA non-real estate secured
Financing Receivable, Past Due [Line Items]
Days Past Due	692	707
Retail loans
Financing Receivable, Past Due [Line Items]
Total	190,373	239,503
Retail loans | Single family residential
Financing Receivable, Past Due [Line Items]
Total	184,049	232,574
Retail loans | Consumer
Financing Receivable, Past Due [Line Items]
Total	6,324	6,929
Retail loans | Current
Financing Receivable, Past Due [Line Items]
Current	190,373	239,487
Retail loans | Current | Single family residential
Financing Receivable, Past Due [Line Items]
Current	184,049	232,559
Retail loans | Current | Consumer
Financing Receivable, Past Due [Line Items]
Current	6,324	6,928
Retail loans | 30-59
Financing Receivable, Past Due [Line Items]
Days Past Due	0	16
Retail loans | 30-59 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	15
Retail loans | 30-59 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	1
Retail loans | 60-89
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 60-89 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 60-89 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90 | Single family residential
Financing Receivable, Past Due [Line Items]
Days Past Due	0	0
Retail loans | 90 | Consumer
Financing Receivable, Past Due [Line Items]
Days Past Due	$ 0	$ 0