Loans and Allowance for Credit Losses - Credit Quality (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2024	Dec. 31, 2023
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 9,482,906	$ 10,675,958
General allowance for loan losses	(36,618)	(44,473)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 9,446,288	$ 10,631,485
Percentage of loans outstanding	100.00%	100.00%
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 148,315	$ 170,640
Percentage of loans outstanding	1.60%	1.60%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 326,455	$ 443,633
Percentage of loans outstanding	3.40%	4.20%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,592,191	$ 1,995,916
Percentage of loans outstanding	16.80%	18.70%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 6,432,367	$ 7,133,968
Percentage of loans outstanding	67.80%	66.80%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 534,508	$ 462,124
Percentage of loans outstanding	5.60%	4.30%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 449,070	$ 469,677
Percentage of loans outstanding	4.80%	4.40%
Bridge
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 6,069,908	$ 6,796,082
General allowance for loan losses	(12,732)	(17,302)
Bridge | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,243	2,308
Bridge | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	85,672	97,309
Bridge | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	721,964	756,353
Bridge | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,974,660	5,781,651
Bridge | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	179,286	82,517
Bridge | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	106,083	75,944
Fixed rate
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	959,022	1,031,830
General allowance for loan losses	(5,415)	(7,884)
Fixed rate | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,227	5,222
Fixed rate | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	30,987	36,021
Fixed rate | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	437,170	449,804
Fixed rate | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	466,402	517,628
Fixed rate | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	19,940	19,965
Fixed rate | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,296	3,190
Construction
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	839,625	1,180,635
General allowance for loan losses	(5,840)	(3,722)
Construction | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	21,019	25,173
Construction | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,521	94,856
Construction | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	147,031	532,730
Construction | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	569,839	355,631
Construction | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	91,895	119,191
Construction | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,320	53,054
Freddie Mac
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		9,500
Freddie Mac | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		2,995
Freddie Mac | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		6,505
SBA 7(a)
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,218,807	1,204,678
General allowance for loan losses	(10,204)	(12,679)
SBA 7(a) | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	13,798	10,627
SBA 7(a) | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	69,428	56,061
SBA 7(a) | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	218,269	172,743
SBA 7(a) | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	353,621	404,102
SBA 7(a) | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	228,658	226,327
SBA 7(a) | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	335,033	334,818
Other
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	395,544	453,233
General allowance for loan losses	(2,427)	(2,886)
Other | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	108,028	127,310
Other | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	135,847	159,386
Other | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	67,757	81,291
Other | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	67,845	68,451
Other | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	14,729	14,124
Other | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,338	$ 2,671