Loans Held for Investment - Internal Risk Grading System under ASC 326 (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	$ 758,874	$ 1,862,085
Year two	1,837,742	3,354,750
Year three	3,184,608	2,140,513
Year four	1,974,365	1,729,763
Year five	1,623,130	1,072,603
Prior	2,947,726	2,152,690
Revolving	789,248	912,643
Revolving Converted to Term During the Period	1,699	11,386
Total	13,117,392	13,236,433
Investor loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	541,743	1,351,350
Year two	1,354,913	2,386,517
Year three	2,301,001	1,437,278
Year four	1,352,833	988,649
Year five	938,701	674,159
Prior	1,909,229	1,374,935
Revolving	13,796	12,877
Revolving Converted to Term During the Period	0	0
Total	8,412,216	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,729,785	2,675,085
Investor loans secured by real estate | CRE non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	124,225	265,901
Year two	272,001	541,994
Year three	501,036	440,351
Year four	482,052	287,580
Year five	278,439	279,238
Prior	972,783	791,477
Revolving	10,779	11,114
Revolving Converted to Term During the Period	0	0
Total	2,641,315	2,617,655
Investor loans secured by real estate | CRE non-owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	2,004	6,669
Year four	8,619	437
Year five	438	2,516
Prior	43,694	29,738
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	54,755	39,360
Investor loans secured by real estate | CRE non-owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	9,732
Year three	25,740	2,045
Year four	1,856	0
Year five	0	516
Prior	5,571	5,218
Revolving	548	559
Revolving Converted to Term During the Period	0	0
Total	33,715	18,070
Investor loans secured by real estate | Multifamily
Financing Receivable, Credit Quality Indicator [Line Items]
Total	5,309,592	5,171,356
Investor loans secured by real estate | Multifamily | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	405,022	1,027,644
Year two	1,005,928	1,677,716
Year three	1,626,859	899,123
Year four	791,365	665,939
Year five	629,525	354,859
Prior	834,458	531,287
Revolving	1,426	420
Revolving Converted to Term During the Period	0	0
Total	5,294,583	5,156,988
Investor loans secured by real estate | Multifamily | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	1,758
Year three	1,755	2,630
Year four	2,629	0
Year five	0	8,649
Prior	9,294	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	13,678	13,037
Investor loans secured by real estate | Multifamily | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	559
Year five	559	772
Prior	772	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	1,331	1,331
Investor loans secured by real estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Total	316,458	321,993
Investor loans secured by real estate | Construction and land | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	12,496	57,309
Year two	76,486	144,759
Year three	133,997	73,313
Year four	53,795	18,625
Year five	12,264	20,531
Prior	26,377	6,672
Revolving	1,043	784
Revolving Converted to Term During the Period	0	0
Total	316,458	321,993
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	56,381	57,331
Investor loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	8,306
Year three	7,409	9,029
Year four	8,991	13,418
Year five	15,383	6,305
Prior	12,485	7,696
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	44,268	44,754
Investor loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	496
Year two	498	1,032
Year three	1,033	1,159
Year four	1,161	1,000
Year five	1,002	373
Prior	1,124	306
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	4,818	4,366
Investor loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	1,220
Year three	1,168	2,959
Year four	2,365	1,091
Year five	1,091	400
Prior	2,671	2,541
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	7,295	8,211
Business loans secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	132,455	344,058
Year two	333,341	516,400
Year three	454,173	422,995
Year four	367,772	446,733
Year five	410,952	267,062
Prior	736,867	529,565
Revolving	9,196	14,518
Revolving Converted to Term During the Period	0	246
Total	2,444,756	2,541,577
Business loans secured by real estate | CRE non-owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one		344,058
Year two		516,400
Year three		422,995
Year four		446,733
Year five		267,062
Prior		529,565
Revolving		14,518
Revolving Converted to Term During the Period		246
Total		2,541,577
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total	73,967	79,595
Business loans secured by real estate | SBA secured by real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	510	3,253
Year two	3,237	7,637
Year three	7,587	12,608
Year four	12,262	16,058
Year five	13,155	8,488
Prior	27,937	23,624
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	64,688	71,668
Business loans secured by real estate | SBA secured by real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	1,200
Year four	1,186	0
Year five	0	137
Prior	1,175	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	2,361	1,337
Business loans secured by real estate | SBA secured by real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	184
Year four	161	1,987
Year five	2,111	1,376
Prior	4,646	3,043
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	6,918	6,590
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total	2,029,984	2,114,050
Business loans secured by real estate | CRE owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	108,753	293,324
Year two	283,401	409,758
Year three	373,667	332,672
Year four	280,703	327,475
Year five	304,938	225,098
Prior	638,844	469,704
Revolving	8,946	14,268
Revolving Converted to Term During the Period	0	246
Total	1,999,252	2,072,545
Business loans secured by real estate | CRE owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	2,190
Year two	2,190	15,917
Year three	0	3,802
Year four	3,792	0
Year five	3,045	4,153
Prior	4,319	201
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	13,346	26,263
Business loans secured by real estate | CRE owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	3,636
Year four	4,143	4,214
Year five	5,884	1,169
Prior	7,109	5,973
Revolving	250	250
Revolving Converted to Term During the Period	0	0
Total	17,386	15,242
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	340,805	347,932
Business loans secured by real estate | Franchise real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	23,192	44,413
Year two	43,635	81,438
Year three	71,269	66,241
Year four	62,905	96,999
Year five	81,819	24,673
Prior	52,837	27,020
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	335,657	340,784
Business loans secured by real estate | Franchise real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	878
Year two	878	1,650
Year three	1,650	2,652
Year four	2,620	0
Year five	0	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	5,148	5,180
Business loans secured by real estate | Franchise real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one		0
Year two		0
Year three		0
Year four		0
Year five		1,968
Prior		0
Revolving		0
Revolving Converted to Term During the Period		0
Total		1,968
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	78,092	155,831
Year two	141,817	444,000
Year three	424,977	266,221
Year four	243,017	280,317
Year five	262,286	97,412
Prior	176,638	120,099
Revolving	741,521	854,568
Revolving Converted to Term During the Period	1,699	11,140
Total	2,070,047	2,229,588
Commercial loans | Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Total	1,656,098	1,768,834
Commercial loans | Commercial and industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	62,622	127,082
Year two	112,135	260,368
Year three	272,585	159,001
Year four	142,706	210,163
Year five	195,658	51,800
Prior	112,930	82,291
Revolving	694,022	801,752
Revolving Converted to Term During the Period	1,699	9,315
Total	1,594,357	1,701,772
Commercial loans | Commercial and industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	735
Year two	655	0
Year three	0	2,331
Year four	2,219	185
Year five	177	1,320
Prior	1,460	243
Revolving	16,377	17,890
Revolving Converted to Term During the Period	0	37
Total	20,888	22,741
Commercial loans | Commercial and industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	1,389	3,310
Year three	2,936	2,737
Year four	3,619	610
Year five	218	1,333
Prior	3,622	2,446
Revolving	29,069	32,858
Revolving Converted to Term During the Period	0	1,027
Total	40,853	44,321
Commercial loans | Franchise non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	399,041	444,797
Commercial loans | Franchise non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	15,398	27,607
Year two	27,216	164,025
Year three	138,064	94,494
Year four	86,988	46,174
Year five	43,294	40,829
Prior	50,126	27,745
Revolving	1,361	1,361
Revolving Converted to Term During the Period	0	502
Total	362,447	402,737
Commercial loans | Franchise non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	7,267
Year three	7,028	2,037
Year four	1,984	230
Year five	216	480
Prior	2,605	2,321
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	11,833	12,335
Commercial loans | Franchise non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	6,690
Year three	2,071	3,706
Year four	3,637	18,425
Year five	18,181	700
Prior	872	204
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	24,761	29,725
Commercial loans | SBA non-real estate secured
Financing Receivable, Credit Quality Indicator [Line Items]
Total	14,908	15,957
Commercial loans | SBA non-real estate secured | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	72	407
Year two	422	2,257
Year three	2,212	1,558
Year four	1,511	2,674
Year five	2,816	610
Prior	4,337	4,449
Revolving	0	0
Revolving Converted to Term During the Period	0	259
Total	11,370	12,214
Commercial loans | SBA non-real estate secured | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	1,574
Year five	1,456	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	1,456	1,574
Commercial loans | SBA non-real estate secured | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	83
Year three	81	357
Year four	353	282
Year five	270	340
Prior	686	400
Revolving	692	707
Revolving Converted to Term During the Period	0	0
Total	2,082	2,169
Retail loans
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	6,584	10,846
Year two	7,671	7,833
Year three	4,457	14,019
Year four	10,743	14,064
Year five	11,191	33,970
Prior	124,992	128,091
Revolving	24,735	30,680
Revolving Converted to Term During the Period	0	0
Total	190,373	239,503
Retail loans | Single family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Total	184,049	232,574
Retail loans | Single family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	6,582	10,794
Year two	7,624	7,714
Year three	4,349	13,982
Year four	10,708	14,039
Year five	11,169	33,968
Prior	120,675	124,248
Revolving	22,662	27,172
Revolving Converted to Term During the Period	0	0
Total	183,769	231,917
Retail loans | Single family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	0
Year five	0	0
Prior	280	657
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	280	657
Retail loans | Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Total	6,324	6,929
Retail loans | Consumer | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	2	52
Year two	47	112
Year three	101	37
Year four	35	25
Year five	22	2
Prior	3,998	3,145
Revolving	2,073	3,508
Revolving Converted to Term During the Period	0	0
Total	6,278	6,881
Retail loans | Consumer | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Year one	0	0
Year two	0	7
Year three	7	0
Year four	0	0
Year five	0	0
Prior	39	41
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	$ 46	$ 48