Loans and Allowance for Credit Losses - Credit Quality (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 10,675,958	$ 9,941,207
General allowance for loan losses	(44,473)	(57,710)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 10,631,485	$ 9,883,497
Percentage of loans outstanding	100.00%	100.00%
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 170,640	$ 201,631
Percentage of loans outstanding	1.60%	2.00%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 443,633	$ 412,711
Percentage of loans outstanding	4.20%	4.20%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,995,916	$ 1,324,844
Percentage of loans outstanding	18.70%	13.30%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 7,133,968	$ 7,525,735
Percentage of loans outstanding	66.80%	75.70%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 462,124	$ 295,449
Percentage of loans outstanding	4.30%	3.00%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 469,677	$ 180,837
Percentage of loans outstanding	4.40%	1.80%
Bridge
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 6,796,082	$ 7,327,946
General allowance for loan losses	(17,302)	(42,979)
Bridge | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,308	717
Bridge | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	97,309	104,606
Bridge | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	756,353	700,835
Bridge | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,781,651	6,331,353
Bridge | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	82,517	167,521
Bridge | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	75,944	22,914
Fixed rate
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,031,830	1,034,626
General allowance for loan losses	(7,884)	(2,397)
Fixed rate | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,222	9,102
Fixed rate | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	36,021	35,459
Fixed rate | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	449,804	386,040
Fixed rate | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	517,628	578,456
Fixed rate | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	19,965	17,056
Fixed rate | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,190	8,513
Construction
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,180,635	428,805
General allowance for loan losses	(3,722)	(325)
Construction | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	25,173	10,817
Construction | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	94,856	12,910
Construction | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	532,730	26,387
Construction | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	355,631	349,085
Construction | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	119,191	24,142
Construction | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	53,054	5,464
Freddie Mac
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	9,500	10,040
Freddie Mac | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,995	3,056
Freddie Mac | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	6,505	6,984
SBA 7(a)
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,204,678	552,260
General allowance for loan losses	(12,679)	(10,801)
SBA 7(a) | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	10,627	7,275
SBA 7(a) | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	56,061	45,366
SBA 7(a) | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	172,743	92,592
SBA 7(a) | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	404,102	189,733
SBA 7(a) | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	226,327	78,577
SBA 7(a) | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	334,818	138,717
Other
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	453,233	587,530
General allowance for loan losses	(2,886)	(1,208)
Other | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	127,310	173,720
Other | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	159,386	214,370
Other | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	81,291	115,934
Other | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	68,451	70,124
Other | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	14,124	8,153
Other | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 2,671	$ 5,229