Consolidated Statements of Cash Flows - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Cash flows from operating activities
Net income (loss) before attribution of noncontrolling interest	$ 61,963	$ 110,311	$ (1,722)
Adjustments to reconcile net income (loss) to net cash provided by operating activities
Provision for (reversal of) credit losses	13,945	(16,500)	88,620
Net premium amortization on securities	8,537	12,596	14,868
Depreciation and amortization	5,883	7,269	9,385
Stock-based compensation expense	5,788	4,782	2,307
Change in cash surrender value of bank owned life insurance	(5,406)	(5,459)	(5,695)
Securities losses (gains), net	3,689	(3,740)	(26,990)
Derivative gains, net	(455)	0	0
Loss (gain) on sale of loans, net	320	(4,276)	0
Net gain on sale of headquarters building	0	(62,387)	0
Net loss (gain) on sale of premises and equipment	0	71	1,729
Deferred taxes and others	5,157	6,000	(11,513)
(Gain) loss on early extinguishment of advances from the FHLB, net	(10,678)	2,488	73
Proceeds from sales and repayments of mortgage loans originated for sale (at fair value)	143,082	20,859	0
Originations and purchases of mortgage loans originated for sale (at fair value)	(286,715)	(35,108)	0
Net changes in operating assets and liabilities
Accrued interest receivable and other assets	(15,348)	(4,432)	(446)
Account payable, accrued liabilities and other liabilities	21,078	34,957	(13,369)
Net cash (used in) provided by operating activities	(49,160)	67,431	57,247
Purchases of investment securities:
Available for sale	(266,667)	(425,864)	(399,202)
Held to maturity	(140,028)	(100,403)	0
Federal Home Loan Bank stock	(38,044)	(4,565)	(9,843)
Equity securities with readily determinable fair value not held for trading	(12,656)	0	(29)
Purchases of investment securities	(457,395)	(530,832)	(409,074)
Maturities, sales, calls, paydowns and redemptions of investment securities:
Available for sale	246,394	446,436	781,983
Held to maturity	15,354	39,695	15,056
Federal Home Loan Bank stock	29,964	22,110	18,742
Equity securities with readily determinable fair value not held for trading	252	23,470	0
Maturities, sales, calls and paydowns of investment securities	291,964	531,711	815,781
Net proceeds from sale of headquarters building	0	132,360	0
Net (increase) decrease in loans	(1,311,608)	93,321	(199,910)
Proceeds from loan portfolio sales	84,029	166,329	71,639
Purchases of premises and equipment	(10,629)	(6,577)	(5,573)
Proceeds from sales of premises and equipment	0	28	13,476
Proceeds from sales of other real estate owned	6,393	16	0
Cash paid in business acquisition, net	0	(1,037)	0
Net cash (used in) provided by investing activities	(1,397,246)	385,319	286,339
Cash flows from financing activities
Net increase in demand, savings and money market accounts	1,022,913	602,950	353,277
Net increase (decrease) in time deposits	390,415	(703,722)	(378,777)
Proceeds from advances from the Federal Home Loan Bank	1,130,000	485,500	750,000
Repayments of advances from the Federal Home Loan Bank	(1,024,322)	(729,618)	(935,073)
Proceeds from issuance of subordinated notes, net of issuance costs	29,146	0	0
Proceeds from issuance of Senior Notes, net of issuance costs	0	0	58,412
Redemption of junior subordinated debentures	0	0	(28,068)
Repurchase of common stock - Class A	(72,060)	(36,332)	0
Dividends paid	(12,230)	0	0
Repurchase of common stock - Class B	0	(9,563)	(69,378)
Common stock surrendered	(1,063)	(2,143)	(917)
Net cash provided by (used in) financing activities	1,462,799	(392,928)	(250,524)
Net increase in cash and cash equivalents and restricted cash	16,393	59,822	93,062
Cash and cash equivalents and restricted cash
Beginning of period	274,208	214,386	121,324
End of period	290,601	274,208	214,386
Supplemental disclosures of cash flow information
Cash paid for Interest	67,295	46,327	73,349
Cash paid for Income taxes	27,537	14,538	10,576
Initial recognition of operating lease right-of-use assets	0	55,670	0
Initial recognition of operating lease liabilities	0	56,024	0
Right-of-use assets obtained in exchange for new lease obligations	8,887	91,797	0
Noncash investing activities:
Mortgage loans held for sale (at fair value) transferred to loans held for investment	96,233	0	0
Loans held for sale (at lower cost or fair value) transferred to loans held for investment)	65,802	0	0
Loans held for investment transferred to loans held for sale	0	256,154	0
Net transfers from premises and equipments to operating lease right-of-use assets	0	69,931	0
Loans transferred to other assets	$ 0	$ 9,400	$ 400