Loans and Allowance for Credit Losses - Credit Quality (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2024	Dec. 31, 2023
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 9,789,038	$ 10,675,958
General allowance for loan losses	(45,873)	(44,473)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 9,743,165	$ 10,631,485
Percentage of loans outstanding	100.00%	100.00%
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 151,766	$ 170,640
Percentage of loans outstanding	1.60%	1.60%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 376,497	$ 443,633
Percentage of loans outstanding	3.80%	4.20%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,572,109	$ 1,995,916
Percentage of loans outstanding	16.10%	18.70%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 6,827,591	$ 7,133,968
Percentage of loans outstanding	69.70%	66.80%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 458,571	$ 462,124
Percentage of loans outstanding	4.70%	4.30%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 402,504	$ 469,677
Percentage of loans outstanding	4.10%	4.40%
Bridge
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 6,297,351	$ 6,796,082
General allowance for loan losses	(10,827)	(17,302)
Bridge | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,259	2,308
Bridge | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	102,161	97,309
Bridge | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	700,851	756,353
Bridge | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,353,476	5,781,651
Bridge | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	108,534	82,517
Bridge | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	30,070	75,944
Fixed rate
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	975,213	1,031,830
General allowance for loan losses	(5,971)	(7,884)
Fixed rate | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,231	5,222
Fixed rate | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	32,393	36,021
Fixed rate | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	441,069	449,804
Fixed rate | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	476,391	517,628
Fixed rate | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	19,833	19,965
Fixed rate | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,296	3,190
Construction
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	890,432	1,180,635
General allowance for loan losses	(11,886)	(3,722)
Construction | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	15,292	25,173
Construction | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	25,152	94,856
Construction | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	128,515	532,730
Construction | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	572,255	355,631
Construction | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	95,989	119,191
Construction | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	53,229	53,054
Freddie Mac
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		9,500
Freddie Mac | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		2,995
Freddie Mac | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		6,505
SBA 7(a)
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,207,102	1,204,678
General allowance for loan losses	(14,596)	(12,679)
SBA 7(a) | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	12,165	10,627
SBA 7(a) | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	72,245	56,061
SBA 7(a) | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	225,117	172,743
SBA 7(a) | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	361,900	404,102
SBA 7(a) | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	229,108	226,327
SBA 7(a) | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	306,567	334,818
Other
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	418,940	453,233
General allowance for loan losses	(2,593)	(2,886)
Other | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	117,819	127,310
Other | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	144,546	159,386
Other | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	76,557	81,291
Other | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	63,569	68,451
Other | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,107	14,124
Other | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 11,342	$ 2,671