Loans and Allowance for Credit Losses - Loan Vintage (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2024	Dec. 31, 2023
Loan classification and delinquency by year of origination
UPB	$ 9,851,877	$ 10,780,189
Current fiscal year	137,115	590,350
Year before current fiscal year	559,231	3,679,345
Two years before current fiscal year	3,378,635	3,680,518
Three years before current fiscal year	3,451,975	582,228
Four years before current fiscal year	464,900	1,038,298
Five or more years before current fiscal year	1,797,182	1,105,219
Total	9,789,038	10,675,958
General allowance for loan losses	(45,873)	(44,473)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	9,743,165	10,631,485
Specific allowance for loan losses including PCD allowance	21,600	57,100
Specific allowance for purchased financial assets with credit deterioration	11,179	21,422
Specific allowance for loan losses	10,371	35,710
Gross write-offs by year of origination
Current fiscal year write-offs		100
Year before current fiscal year write-offs	494	950
Two years before current fiscal year write-offs	873	3,236
Three years before current fiscal year write-offs	35	258
Four years before current fiscal year write-offs	201	360
Five or more years before current fiscal year write-offs	4,150	25,731
Total write-offs	5,753	30,635
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	9,014,343	9,632,399
Current fiscal year	137,108	574,507
Year before current fiscal year	540,691	3,351,046
Two years before current fiscal year	3,116,363	3,409,643
Three years before current fiscal year	3,060,184	495,433
Four years before current fiscal year	413,838	881,868
Five or more years before current fiscal year	1,708,790	875,348
Total	8,976,974	9,587,845
30-59 Days Past Due
Loan classification and delinquency by year of origination
UPB	199,535	172,355
Current fiscal year	7	582
Year before current fiscal year	3,490	59,988
Two years before current fiscal year	141,674	80,684
Three years before current fiscal year	38,498	510
Four years before current fiscal year	1,650	22,586
Five or more years before current fiscal year	11,781	7,148
Total	197,100	171,498
60+ Days Past Due
Loan classification and delinquency by year of origination
UPB	637,999	975,435
Current fiscal year		15,261
Year before current fiscal year	15,050	268,311
Two years before current fiscal year	120,598	190,191
Three years before current fiscal year	353,293	86,285
Four years before current fiscal year	49,412	133,844
Five or more years before current fiscal year	76,611	222,723
Total	614,964	916,615
Bridge
Loan classification and delinquency by year of origination
UPB	6,318,551	6,837,816
Current fiscal year	86,682	323,648
Year before current fiscal year	290,638	2,956,697
Two years before current fiscal year	2,775,887	2,949,521
Three years before current fiscal year	2,817,308	288,647
Four years before current fiscal year	206,891	166,266
Five or more years before current fiscal year	119,945	111,303
Total	6,297,351	6,796,082
General allowance for loan losses	(10,827)	(17,302)
Specific allowance for loan losses	2,354	18,939
Bridge | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	5,705,330	6,186,367
Bridge | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	139,459	87,163
Bridge | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	452,562	522,552
Fixed rate
Loan classification and delinquency by year of origination
UPB	973,437	1,032,641
Current fiscal year		4,007
Year before current fiscal year	4,007	110,800
Two years before current fiscal year	110,737	207,510
Three years before current fiscal year	181,596	90,794
Four years before current fiscal year	90,433	318,077
Five or more years before current fiscal year	588,440	300,642
Total	975,213	1,031,830
General allowance for loan losses	(5,971)	(7,884)
Specific allowance for loan losses	1,293	5,714
Fixed rate | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	952,931	986,755
Fixed rate | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total		21,798
Fixed rate | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	22,282	23,277
Construction
Loan classification and delinquency by year of origination
UPB	904,162	1,212,526
Current fiscal year		108,218
Year before current fiscal year	111,323	253,100
Two years before current fiscal year	138,755	182,920
Three years before current fiscal year	138,497	73,370
Four years before current fiscal year	47,469	434,151
Five or more years before current fiscal year	454,388	128,876
Total	890,432	1,180,635
General allowance for loan losses	(11,886)	(3,722)
Specific allowance for purchased financial assets with credit deterioration	11,179	21,422
Specific allowance for loan losses	690	5,726
Construction | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	730,640	782,123
Construction | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	31,891	49,694
Construction | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	127,901	348,818
Freddie Mac
Loan classification and delinquency by year of origination
UPB		9,719
Two years before current fiscal year		3,810
Three years before current fiscal year		5,690
Total		9,500
Freddie Mac | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total		9,500
SBA 7(a)
Loan classification and delinquency by year of origination
UPB	1,233,906	1,230,959
Current fiscal year	44,096	151,878
Year before current fiscal year	150,457	353,871
Two years before current fiscal year	348,424	318,208
Three years before current fiscal year	302,761	115,019
Four years before current fiscal year	111,453	76,080
Five or more years before current fiscal year	249,911	189,622
Total	1,207,102	1,204,678
General allowance for loan losses	(14,596)	(12,679)
Specific allowance for loan losses	5,983	5,188
SBA 7(a) | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,173,270	1,179,231
SBA 7(a) | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	25,448	8,619
SBA 7(a) | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	8,384	16,828
Other
Loan classification and delinquency by year of origination
UPB	421,821	456,528
Current fiscal year	6,337	2,599
Year before current fiscal year	2,806	4,877
Two years before current fiscal year	4,832	18,549
Three years before current fiscal year	11,813	8,708
Four years before current fiscal year	8,654	43,724
Five or more years before current fiscal year	384,498	374,776
Total	418,940	453,233
General allowance for loan losses	(2,593)	(2,886)
Specific allowance for loan losses	51	143
Other | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	414,803	443,869
Other | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	302	4,224
Other | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 3,835	$ 5,140