Allowance for Loan Losses - Credit Risk Quality Indicators (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	$ 5,409,440	$ 5,842,337
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	5,283,998	5,634,991
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	74,086	118,813
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	40,464	61,308
Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	10,892	27,225
Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	4,006,392	4,424,031
Real estate loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	3,943,493	4,303,765
Real estate loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	41,634	76,379
Real estate loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	19,870	39,918
Real estate loans | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	1,395	3,969
Real estate loans | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Non-owner occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	1,540,590	1,749,839
Real estate loans | Non-owner occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	1,499,100	1,694,004
Real estate loans | Non-owner occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	34,205	46,872
Real estate loans | Non-owner occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	5,890	4,994
Real estate loans | Non-owner occupied | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	1,395	3,969
Real estate loans | Non-owner occupied | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Multi-family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	514,679	737,696
Real estate loans | Multi-family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	514,679	726,356
Real estate loans | Multi-family residential | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Multi-family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	11,340
Real estate loans | Multi-family residential | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Multi-family residential | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Land development and construction loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	327,246	349,800
Real estate loans | Land development and construction loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	327,246	342,636
Real estate loans | Land development and construction loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	7,164
Real estate loans | Land development and construction loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Land development and construction loans | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Land development and construction loans | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	2,382,515	2,837,335
Real estate loans | Commercial real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	2,341,025	2,762,996
Real estate loans | Commercial real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	34,205	54,036
Real estate loans | Commercial real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	5,890	16,334
Real estate loans | Commercial real estate | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	1,395	3,969
Real estate loans | Commercial real estate | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Single-family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	661,339	639,569	$ 539,102
Real estate loans | Single-family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	656,118	628,902
Real estate loans | Single-family residential | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Single-family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	5,221	10,667
Real estate loans | Single-family residential | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Single-family residential | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Owner-occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	962,538	947,127
Real estate loans | Owner-occupied | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	946,350	911,867
Real estate loans | Owner-occupied | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	7,429	22,343
Real estate loans | Owner-occupied | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	8,759	12,917
Real estate loans | Owner-occupied | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Real estate loans | Owner-occupied | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	965,673	1,154,550
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	903,400	1,067,708
Commercial loans | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	32,452	42,434
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	20,324	21,152
Commercial loans | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	9,497	23,256
Commercial loans | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Loans to financial institutions and acceptances
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	13,710	16,636
Loans to financial institutions and acceptances | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	13,710	16,636
Loans to financial institutions and acceptances | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Loans to financial institutions and acceptances | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Loans to financial institutions and acceptances | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Loans to financial institutions and acceptances | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Consumer loans and overdrafts
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	423,665	247,120	$ 88,466
Consumer loans and overdrafts | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	423,395	246,882
Consumer loans and overdrafts | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Consumer loans and overdrafts | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	270	238
Consumer loans and overdrafts | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	0	0
Consumer loans and overdrafts | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loan Balance	$ 0	$ 0