Loans and Allowance for Credit Losses - Loan Vintage (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022
Loan classification and delinquency by year of origination
UPB	$ 10,780,189	$ 10,047,316
Current fiscal year	590,350	3,183,566
Year before current fiscal year	3,679,345	3,830,142
Two years before current fiscal year	3,680,518	511,122
Three years before current fiscal year	582,228	1,133,290
Four years before current fiscal year	1,038,298	417,697
Five or more years before current fiscal year	1,105,219	865,390
Total	10,675,958	9,941,207
General allowance for loan losses	(44,473)	(57,710)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	10,631,485	9,883,497
Specific allowance for loan losses including PCD allowance	57,100	32,800
Specific allowance for purchased financial assets with credit deterioration	21,422	15,972
Specific allowance for loan losses	35,710	16,837
Gross write-offs by year of origination
Current fiscal year write-offs	100
Year before current fiscal year write-offs	950
Two years before current fiscal year write-offs	3,236
Three years before current fiscal year write-offs	258
Four years before current fiscal year write-offs	360
Five or more years before current fiscal year write-offs	25,731
Total write-offs	30,635
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	9,632,399	9,663,456
Current fiscal year	574,507	3,098,103
Year before current fiscal year	3,351,046	3,826,008
Two years before current fiscal year	3,409,643	500,807
Three years before current fiscal year	495,433	1,060,723
Four years before current fiscal year	881,868	298,208
Five or more years before current fiscal year	875,348	810,085
Total	9,587,845	9,593,934
30-59 Days Past Due
Loan classification and delinquency by year of origination
UPB	172,355	111,992
Current fiscal year	582	85,403
Year before current fiscal year	59,988	3,483
Two years before current fiscal year	80,684	1,634
Three years before current fiscal year	510	6,654
Four years before current fiscal year	22,586	11,190
Five or more years before current fiscal year	7,148	1,948
Total	171,498	110,312
60+ Days Past Due
Loan classification and delinquency by year of origination
UPB	975,435	271,868
Current fiscal year	15,261	60
Year before current fiscal year	268,311	651
Two years before current fiscal year	190,191	8,681
Three years before current fiscal year	86,285	65,913
Four years before current fiscal year	133,844	108,299
Five or more years before current fiscal year	222,723	53,357
Total	916,615	236,961
Bridge
Loan classification and delinquency by year of origination
UPB	6,837,816	7,381,963
Current fiscal year	323,648	2,942,695
Year before current fiscal year	2,956,697	3,575,213
Two years before current fiscal year	2,949,521	355,647
Three years before current fiscal year	288,647	288,957
Four years before current fiscal year	166,266	137,463
Five or more years before current fiscal year	111,303	27,971
Total	6,796,082	7,327,946
General allowance for loan losses	(17,302)	(42,979)
Specific allowance for loan losses	18,939	6,926
Bridge | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	6,186,367	7,120,162
Bridge | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	87,163	94,823
Bridge | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	522,552	112,961
Fixed rate
Loan classification and delinquency by year of origination
UPB	1,032,641	1,032,199
Current fiscal year	4,007	96,897
Year before current fiscal year	110,800	154,077
Two years before current fiscal year	207,510	92,080
Three years before current fiscal year	90,794	343,500
Four years before current fiscal year	318,077	134,666
Five or more years before current fiscal year	300,642	213,406
Total	1,031,830	1,034,626
General allowance for loan losses	(7,884)	(2,397)
Specific allowance for loan losses	5,714	4,134
Fixed rate | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	986,755	993,832
Fixed rate | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	21,798	8,101
Fixed rate | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	23,277	32,693
Construction
Loan classification and delinquency by year of origination
UPB	1,212,526	448,923
Current fiscal year	108,218	27,532
Year before current fiscal year	253,100
Two years before current fiscal year	182,920	10,000
Three years before current fiscal year	73,370	348,622
Four years before current fiscal year	434,151	42,651
Five or more years before current fiscal year	128,876
Total	1,180,635	428,805
General allowance for loan losses	(3,722)	(325)
Specific allowance for purchased financial assets with credit deterioration	21,422	15,972
Specific allowance for loan losses	5,726	1,037
Construction | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	782,123	372,812
Construction | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	49,694
Construction | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	348,818	55,993
Freddie Mac
Loan classification and delinquency by year of origination
UPB	9,719	9,932
Year before current fiscal year		3,891
Two years before current fiscal year	3,810	6,149
Three years before current fiscal year	5,690
Total	9,500	10,040
Freddie Mac | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	9,500	6,947
Freddie Mac | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total		3,093
SBA 7(a)
Loan classification and delinquency by year of origination
UPB	1,230,959	580,576
Current fiscal year	151,878	110,549
Year before current fiscal year	353,871	79,946
Two years before current fiscal year	318,208	36,853
Three years before current fiscal year	115,019	77,449
Four years before current fiscal year	76,080	89,085
Five or more years before current fiscal year	189,622	158,378
Total	1,204,678	552,260
General allowance for loan losses	(12,679)	(10,801)
Specific allowance for loan losses	5,188	3,498
SBA 7(a) | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,179,231	541,378
SBA 7(a) | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	8,619	6,690
SBA 7(a) | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	16,828	4,192
Other
Loan classification and delinquency by year of origination
UPB	456,528	593,723
Current fiscal year	2,599	5,893
Year before current fiscal year	4,877	17,015
Two years before current fiscal year	18,549	10,393
Three years before current fiscal year	8,708	74,762
Four years before current fiscal year	43,724	13,832
Five or more years before current fiscal year	374,776	465,635
Total	453,233	587,530
General allowance for loan losses	(2,886)	(1,208)
Specific allowance for loan losses	143	1,242
Other | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	443,869	558,803
Other | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	4,224	698
Other | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 5,140	$ 28,029