Loans and Allowance for Credit Losses - Loan Vintage (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2024	Dec. 31, 2023
Loan classification and delinquency by year of origination
UPB	$ 9,527,115	$ 10,780,189
Current fiscal year	331,591	590,350
Year before current fiscal year	514,620	3,679,345
Two years before current fiscal year	3,285,929	3,680,518
Three years before current fiscal year	3,219,388	582,228
Four years before current fiscal year	397,423	1,038,298
Five or more years before current fiscal year	1,733,955	1,105,219
Total	9,482,906	10,675,958
General allowance for loan losses	(36,618)	(44,473)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	9,446,288	10,631,485
Specific allowance for loan losses including PCD allowance	8,200	57,100
Specific allowance for purchased financial assets with credit deterioration	1,889	21,422
Specific allowance for loan losses	6,325	35,710
Gross write-offs by year of origination
Current fiscal year write-offs		100
Year before current fiscal year write-offs	1,273	950
Two years before current fiscal year write-offs	1,492	3,236
Three years before current fiscal year write-offs	2,890	258
Four years before current fiscal year write-offs	533	360
Five or more years before current fiscal year write-offs	5,511	25,731
Total write-offs	11,699	30,635
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	9,103,724	9,632,399
Current fiscal year	330,985	574,507
Year before current fiscal year	497,155	3,351,046
Two years before current fiscal year	3,168,095	3,409,643
Three years before current fiscal year	3,029,683	495,433
Four years before current fiscal year	380,511	881,868
Five or more years before current fiscal year	1,664,385	875,348
Total	9,070,814	9,587,845
30-59 Days Past Due
Loan classification and delinquency by year of origination
UPB	45,971	172,355
Current fiscal year	505	582
Year before current fiscal year	124	59,988
Two years before current fiscal year	16,759	80,684
Three years before current fiscal year	13,408	510
Four years before current fiscal year		22,586
Five or more years before current fiscal year	15,112	7,148
Total	45,908	171,498
60+ Days Past Due
Loan classification and delinquency by year of origination
UPB	377,420	975,435
Current fiscal year	101	15,261
Year before current fiscal year	17,341	268,311
Two years before current fiscal year	101,075	190,191
Three years before current fiscal year	176,297	86,285
Four years before current fiscal year	16,912	133,844
Five or more years before current fiscal year	54,458	222,723
Total	366,184	916,615
Bridge
Loan classification and delinquency by year of origination
UPB	6,086,253	6,837,816
Current fiscal year	231,149	323,648
Year before current fiscal year	255,672	2,956,697
Two years before current fiscal year	2,669,676	2,949,521
Three years before current fiscal year	2,643,443	288,647
Four years before current fiscal year	176,563	166,266
Five or more years before current fiscal year	93,405	111,303
Total	6,069,908	6,796,082
General allowance for loan losses	(12,732)	(17,302)
Specific allowance for loan losses	1,666	18,939
Bridge | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	5,752,441	6,186,367
Bridge | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	25,074	87,163
Bridge | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	292,393	522,552
Fixed rate
Loan classification and delinquency by year of origination
UPB	957,283	1,032,641
Current fiscal year		4,007
Year before current fiscal year		110,800
Two years before current fiscal year	110,625	207,510
Three years before current fiscal year	181,427	90,794
Four years before current fiscal year	90,094	318,077
Five or more years before current fiscal year	576,876	300,642
Total	959,022	1,031,830
General allowance for loan losses	(5,415)	(7,884)
Specific allowance for loan losses	1,065	5,714
Fixed rate | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	933,012	986,755
Fixed rate | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	3,500	21,798
Fixed rate | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	22,510	23,277
Construction
Loan classification and delinquency by year of origination
UPB	842,854	1,212,526
Current fiscal year		108,218
Year before current fiscal year	107,206	253,100
Two years before current fiscal year	156,856	182,920
Three years before current fiscal year	103,290	73,370
Four years before current fiscal year	12,866	434,151
Five or more years before current fiscal year	459,407	128,876
Total	839,625	1,180,635
General allowance for loan losses	(5,840)	(3,722)
Specific allowance for purchased financial assets with credit deterioration	1,889	21,422
Specific allowance for loan losses	181	5,726
Construction | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	803,198	782,123
Construction | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	15,182	49,694
Construction | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	21,245	348,818
Freddie Mac
Loan classification and delinquency by year of origination
UPB		9,719
Two years before current fiscal year		3,810
Three years before current fiscal year		5,690
Total		9,500
Freddie Mac | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total		9,500
SBA 7(a)
Loan classification and delinquency by year of origination
UPB	1,242,458	1,230,959
Current fiscal year	94,112	151,878
Year before current fiscal year	148,941	353,871
Two years before current fiscal year	344,158	318,208
Three years before current fiscal year	283,853	115,019
Four years before current fiscal year	109,300	76,080
Five or more years before current fiscal year	238,443	189,622
Total	1,218,807	1,204,678
General allowance for loan losses	(10,204)	(12,679)
Specific allowance for loan losses	3,362	5,188
SBA 7(a) | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,192,886	1,179,231
SBA 7(a) | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	262	8,619
SBA 7(a) | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	25,659	16,828
Other
Loan classification and delinquency by year of origination
UPB	398,267	456,528
Current fiscal year	6,330	2,599
Year before current fiscal year	2,801	4,877
Two years before current fiscal year	4,614	18,549
Three years before current fiscal year	7,375	8,708
Four years before current fiscal year	8,600	43,724
Five or more years before current fiscal year	365,824	374,776
Total	395,544	453,233
General allowance for loan losses	(2,427)	(2,886)
Specific allowance for loan losses	51	143
Other | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	389,277	443,869
Other | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	1,890	4,224
Other | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 4,377	$ 5,140