Allowance for Credit Losses - Amortized Cost of Loans by Estimated LTV and FICO Bands (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	$ 758,874	$ 1,862,085
Year two	1,837,742	3,354,750
Year three	3,184,608	2,140,513
Year four	1,974,365	1,729,763
Year five	1,623,130	1,072,603
Prior	2,947,726	2,152,690
Revolving	789,248	912,643
Revolving Converted to Term During the Period	1,699	11,386
Total	13,117,392	13,236,433
Investor loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	541,743	1,351,350
Year two	1,354,913	2,386,517
Year three	2,301,001	1,437,278
Year four	1,352,833	988,649
Year five	938,701	674,159
Prior	1,909,229	1,374,935
Revolving	13,796	12,877
Revolving Converted to Term During the Period	0	0
Total	8,412,216	8,225,765
Investor loans secured by real estate | CRE non-owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,729,785	2,675,085
Investor loans secured by real estate | CRE non-owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	64,538	138,007
Year two	130,003	229,272
Year three	213,339	182,385
Year four	220,420	136,355
Year five	130,687	189,848
Prior	709,045	588,230
Revolving	10,779	11,114
Revolving Converted to Term During the Period	0	0
Total	1,478,811	1,475,211
Investor loans secured by real estate | CRE non-owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	31,752	101,434
Year two	116,313	217,210
Year three	197,381	92,015
Year four	105,577	130,024
Year five	130,405	78,470
Prior	266,760	204,161
Revolving	548	559
Revolving Converted to Term During the Period	0	0
Total	848,736	823,873
Investor loans secured by real estate | CRE non-owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	27,935	26,460
Year two	25,685	102,494
Year three	97,297	169,878
Year four	153,854	18,876
Year five	13,411	13,952
Prior	44,463	29,506
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	362,645	361,166
Investor loans secured by real estate | CRE non-owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	2,750
Year three	20,763	4,787
Year four	12,676	2,762
Year five	4,374	0
Prior	1,780	4,536
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	39,593	14,835
Investor loans secured by real estate | Multifamily
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	5,309,592	5,171,356
Investor loans secured by real estate | Multifamily | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	59,772	218,833
Year two	219,912	345,519
Year three	344,043	294,464
Year four	261,109	233,997
Year five	231,004	84,530
Prior	356,028	269,906
Revolving	1,426	0
Revolving Converted to Term During the Period	0	0
Total	1,473,294	1,447,249
Investor loans secured by real estate | Multifamily | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	173,282	381,737
Year two	389,105	731,408
Year three	716,453	381,282
Year four	359,851	215,170
Year five	211,357	152,066
Prior	328,477	189,151
Revolving	0	420
Revolving Converted to Term During the Period	0	0
Total	2,178,525	2,051,234
Investor loans secured by real estate | Multifamily | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	169,340	427,074
Year two	396,911	583,078
Year three	553,451	215,389
Year four	162,499	215,452
Year five	185,854	127,684
Prior	155,480	66,457
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	1,623,535	1,635,134
Investor loans secured by real estate | Multifamily | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	2,628	0
Year two	0	19,469
Year three	14,667	10,618
Year four	10,535	1,879
Year five	1,869	0
Prior	4,539	5,773
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	34,238	37,739
Investor loans secured by real estate | Construction and land
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	316,458	321,993
Investor loans secured by real estate | Construction and land | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	12,496	57,309
Year two	76,486	105,308
Year three	106,181	36,068
Year four	27,368	18,625
Year five	12,264	20,531
Prior	26,377	6,672
Revolving	1,043	784
Revolving Converted to Term During the Period	0	0
Total	262,215	245,297
Investor loans secured by real estate | Construction and land | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	36,113
Year three	19,603	23,770
Year four	18,590	0
Year five	0	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	38,193	59,883
Investor loans secured by real estate | Construction and land | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	3,338
Year three	8,213	13,475
Year four	7,837	0
Year five	0	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	16,050	16,813
Investor loans secured by real estate | Construction and land | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	0
Year five	0	0
Prior	0	0
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	0	0
Investor loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	56,381	57,331
Investor loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	2,066
Year three	102	649
Year four	645	673
Year five	669	317
Prior	1,088	778
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	2,504	4,483
Investor loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	2,427
Year three	2,420	1,639
Year four	1,633	4,008
Year five	3,991	879
Prior	5,449	4,354
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	13,493	13,307
Investor loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	3,897
Year three	3,888	3,882
Year four	4,641	3,482
Year five	3,477	4,519
Prior	6,619	1,884
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	18,625	17,664
Investor loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	496
Year two	498	2,168
Year three	3,200	6,977
Year four	5,598	7,346
Year five	9,339	1,363
Prior	3,124	3,527
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	21,759	21,877
Business loans secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	132,455	344,058
Year two	333,341	516,400
Year three	454,173	422,995
Year four	367,772	446,733
Year five	410,952	267,062
Prior	736,867	529,565
Revolving	9,196	14,518
Revolving Converted to Term During the Period	0	246
Total	2,444,756	2,541,577
Business loans secured by real estate | SBA secured by real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	73,967	79,595
Business loans secured by real estate | SBA secured by real estate | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	510	1,825
Year two	1,813	1,626
Year three	1,601	5,332
Year four	5,265	5,495
Year five	5,442	3,615
Prior	16,620	13,582
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	31,251	31,475
Business loans secured by real estate | SBA secured by real estate | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	246
Year two	247	513
Year three	512	1,795
Year four	1,740	1,094
Year five	999	3,586
Prior	8,073	5,448
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	11,571	12,682
Business loans secured by real estate | SBA secured by real estate | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	264
Year two	263	3,142
Year three	3,129	1,515
Year four	1,293	3,968
Year five	3,695	1,586
Prior	4,824	4,043
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	13,204	14,518
Business loans secured by real estate | SBA secured by real estate | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	918
Year two	914	2,356
Year three	2,345	5,350
Year four	5,311	7,488
Year five	5,130	1,214
Prior	4,241	3,594
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	17,941	20,920
Business loans secured by real estate | CRE owner-occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	2,029,984	2,114,050
Business loans secured by real estate | CRE owner-occupied | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	41,757	96,803
Year two	100,507	160,605
Year three	148,303	157,868
Year four	122,790	179,791
Year five	173,168	131,795
Prior	433,778	328,188
Revolving	9,196	14,518
Revolving Converted to Term During the Period	0	246
Total	1,029,499	1,069,814
Business loans secured by real estate | CRE owner-occupied | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	28,552	72,044
Year two	64,442	91,028
Year three	75,112	98,176
Year four	87,262	94,712
Year five	101,484	65,120
Prior	131,991	90,548
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	488,843	511,628
Business loans secured by real estate | CRE owner-occupied | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	17,278	71,692
Year two	71,694	152,920
Year three	143,725	79,106
Year four	72,734	43,832
Year five	25,938	31,303
Prior	63,635	31,493
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	395,004	410,346
Business loans secured by real estate | CRE owner-occupied | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	21,166	54,975
Year two	48,948	21,122
Year three	6,527	4,960
Year four	5,852	13,354
Year five	13,277	2,202
Prior	20,868	25,649
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	116,638	122,262
Business loans secured by real estate | Franchise real estate secured
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	340,805	347,932
Business loans secured by real estate | Franchise real estate secured | 55% and below
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	1,884	20,801
Year two	20,715	10,470
Year three	9,853	13,864
Year four	15,109	20,956
Year five	13,479	9,189
Prior	25,410	16,213
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	86,450	91,493
Business loans secured by real estate | Franchise real estate secured | >55-65%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	2,689
Year two	2,666	9,955
Year three	9,905	16,001
Year four	14,514	19,102
Year five	14,248	6,855
Prior	7,615	2,333
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	48,948	56,935
Business loans secured by real estate | Franchise real estate secured | >65-75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	11,298	19,349
Year two	15,452	51,719
Year three	42,286	23,258
Year four	17,909	9,153
Year five	14,740	10,597
Prior	16,607	7,236
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	118,292	121,312
Business loans secured by real estate | Franchise real estate secured | Greater than 75%
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	10,010	2,452
Year two	5,680	10,944
Year three	10,875	15,770
Year four	17,993	47,788
Year five	39,352	0
Prior	3,205	1,238
Revolving	0	0
Revolving Converted to Term During the Period	0	0
Total	87,115	78,192
Retail loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	6,584	10,846
Year two	7,671	7,833
Year three	4,457	14,019
Year four	10,743	14,064
Year five	11,191	33,970
Prior	124,992	128,091
Revolving	24,735	30,680
Revolving Converted to Term During the Period	0	0
Total	190,373	239,503
Retail loans | Single family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	184,049	232,574
Retail loans | Single family residential | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	6,582	10,794
Year two	7,624	6,531
Year three	3,480	12,679
Year four	10,341	8,846
Year five	6,559	28,222
Prior	78,694	81,838
Revolving	15,850	19,588
Revolving Converted to Term During the Period	0	0
Total	129,130	168,498
Retail loans | Single family residential | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	1,183
Year three	869	1,303
Year four	367	4,732
Year five	4,119	2,614
Prior	14,654	15,624
Revolving	5,915	6,685
Revolving Converted to Term During the Period	0	0
Total	25,924	32,141
Retail loans | Single family residential | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	461
Year five	491	3,132
Prior	9,561	7,473
Revolving	863	864
Revolving Converted to Term During the Period	0	0
Total	10,915	11,930
Retail loans | Single family residential | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	0
Year five	0	0
Prior	18,046	19,970
Revolving	34	35
Revolving Converted to Term During the Period	0	0
Total	18,080	20,005
Retail loans | Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	6,324	6,929
Retail loans | Consumer | Greater than 740
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	2	52
Year two	47	69
Year three	63	31
Year four	30	22
Year five	19	1
Prior	3,476	2,609
Revolving	531	2,198
Revolving Converted to Term During the Period	0	0
Total	4,168	4,982
Retail loans | Consumer | >680 - 740
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	35
Year three	32	6
Year four	5	3
Year five	3	0
Prior	459	469
Revolving	1,461	1,227
Revolving Converted to Term During the Period	0	0
Total	1,960	1,740
Retail loans | Consumer | >580 - 680
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	15
Year three	13	0
Year four	0	0
Year five	0	1
Prior	90	95
Revolving	57	56
Revolving Converted to Term During the Period	0	0
Total	160	167
Retail loans | Consumer | Less than 580
Financing Receivable, Allowance for Credit Loss [Line Items]
Year one	0	0
Year two	0	0
Year three	0	0
Year four	0	0
Year five	0	0
Prior	12	13
Revolving	24	27
Revolving Converted to Term During the Period	0	0
Total	$ 36	$ 40