Loans and Allowance for Credit Losses - Delinquency (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2024	Dec. 31, 2023	Mar. 31, 2023	Dec. 31, 2022
Loan delinquency information
Total Loans Carrying Value	$ 9,789,038	$ 10,675,958
Non-Accrual Loans	455,718	634,001
90+ Days Past Due but Accruing	48,956	82,821
General allowance for loan losses	(45,873)	(44,473)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 9,743,165	$ 10,631,485
Percentage of loans outstanding	100.00%	100.00%
Percentage of outstanding, Non-Accrual Loans	4.70%	5.90%
Percentage of outstanding, 90+Days Past Due Accruing	0.50%	0.80%
Specific allowance for loan losses	$ 67,423	$ 101,605	$ 67,780	$ 90,519
Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	$ 8,976,974	$ 9,587,845
Percentage of loans outstanding	91.70%	89.80%
30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 197,100	$ 171,498
Percentage of loans outstanding	2.00%	1.60%
60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 614,964	$ 916,615
Percentage of loans outstanding	6.30%	8.60%
Bridge
Loan delinquency information
Total Loans Carrying Value	$ 6,297,351	$ 6,796,082
Non-Accrual Loans	306,049	339,073
General allowance for loan losses	(10,827)	(17,302)
Specific allowance for loan losses	13,181	36,241	40,319	49,905
Bridge | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	5,705,330	6,186,367
Bridge | 30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value	139,459	87,163
Bridge | 60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	452,562	522,552
Fixed rate
Loan delinquency information
Total Loans Carrying Value	975,213	1,031,830
Non-Accrual Loans	22,283	13,928
General allowance for loan losses	(5,971)	(7,884)
Specific allowance for loan losses	7,264	13,598	9,085	6,531
Fixed rate | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	952,931	986,755
Fixed rate | 30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value		21,798
Fixed rate | 60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	22,282	23,277
Construction
Loan delinquency information
Total Loans Carrying Value	890,432	1,180,635
Non-Accrual Loans	76,861	241,751
90+ Days Past Due but Accruing	48,956	82,781
General allowance for loan losses	(11,886)	(3,722)
Specific allowance for loan losses	23,755	30,870	373	17,334
Construction | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	730,640	782,123
Construction | 30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value	31,891	49,694
Construction | 60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	127,901	348,818
Freddie Mac
Loan delinquency information
Total Loans Carrying Value		9,500
Non-Accrual Loans		2,695
Freddie Mac | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value		9,500
SBA 7(a)
Loan delinquency information
Total Loans Carrying Value	1,207,102	1,204,678
Non-Accrual Loans	36,190	30,549
90+ Days Past Due but Accruing		40
General allowance for loan losses	(14,596)	(12,679)
Specific allowance for loan losses	20,579	17,867	15,110	14,299
SBA 7(a) | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,173,270	1,179,231
SBA 7(a) | 30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value	25,448	8,619
SBA 7(a) | 60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	8,384	16,828
Other
Loan delinquency information
Total Loans Carrying Value	418,940	453,233
Non-Accrual Loans	14,335	6,005
General allowance for loan losses	(2,593)	(2,886)
Specific allowance for loan losses	2,644	3,029	$ 2,893	$ 2,450
Other | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	414,803	443,869
Other | 30-59 Days Past Due
Loan delinquency information
Total Loans Carrying Value	302	4,224
Other | 60+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 3,835	$ 5,140