Loans and Allowance for Credit Losses - Loan Vintage (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Loan classification and delinquency by year of origination
UPB	$ 4,571,128	$ 4,102,035
Current fiscal year	606,835	522,961
Year before current fiscal year	529,003	1,229,069
Two years before current fiscal year	1,208,044	735,604
Three years before current fiscal year	699,588	243,042
Four years before current fiscal year	224,498	110,314
Five or more years before current fiscal year	1,249,098	1,211,980
Total	4,517,066	4,052,970
General allowance for loan losses	(27,403)	(29,539)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	4,489,663	4,023,431
Specific allowance for loan losses	18,200	17,200
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	4,351,956	3,904,294
Current fiscal year	606,638	516,474
Year before current fiscal year	512,632	1,221,227
Two years before current fiscal year	1,178,806	707,068
Three years before current fiscal year	632,881	203,331
Four years before current fiscal year	205,280	100,003
Five or more years before current fiscal year	1,178,466	1,125,100
Total	4,314,703	3,873,203
30-59 Days Past Due
Loan classification and delinquency by year of origination
UPB	79,654	38,836
Current fiscal year		5,812
Year before current fiscal year	15,711	5,191
Two years before current fiscal year	24,790	15,097
Three years before current fiscal year	21,139	401
Four years before current fiscal year	4,841	2
Five or more years before current fiscal year	12,081	11,933
Total	78,562	38,436
60+ Days Past Due
Loan classification and delinquency by year of origination
UPB	139,518	158,905
Current fiscal year	197	675
Year before current fiscal year	660	2,651
Two years before current fiscal year	4,448	13,439
Three years before current fiscal year	45,568	39,310
Four years before current fiscal year	14,377	10,309
Five or more years before current fiscal year	58,551	74,947
Total	123,801	141,331
Originated Transitional loans
Loan classification and delinquency by year of origination
UPB	1,889,158	1,328,395
Current fiscal year	599,301	385,183
Year before current fiscal year	397,349	583,593
Two years before current fiscal year	570,278	306,971
Three years before current fiscal year	277,650	23,783
Four years before current fiscal year	13,292	18,480
Five or more years before current fiscal year	15,294	1,064
Total	1,873,164	1,319,074
General allowance for loan losses	(14,848)	(14,995)
Originated Transitional loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,794,270	1,281,579
Originated Transitional loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	30,315	17,713
Originated Transitional loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	48,579	19,782
Originated SBC loans
Loan classification and delinquency by year of origination
UPB	1,035,280	1,052,705
Current fiscal year		66,715
Year before current fiscal year	62,606	486,033
Two years before current fiscal year	479,979	237,313
Three years before current fiscal year	235,716	110,354
Four years before current fiscal year	105,628	43,696
Five or more years before current fiscal year	155,608	112,444
Total	1,039,537	1,056,555
General allowance for loan losses	(2,729)	(2,640)
Originated SBC loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	993,570	1,000,878
Originated SBC loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	16,953	6,591
Originated SBC loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	29,014	49,086
Acquired loans
Loan classification and delinquency by year of origination
UPB	997,864	1,053,679
Current fiscal year		21,414
Year before current fiscal year	21,042	40,572
Two years before current fiscal year	40,618	42,167
Three years before current fiscal year	41,301	38,649
Four years before current fiscal year	37,507	19,533
Five or more years before current fiscal year	849,833	883,774
Total	990,301	1,046,109
General allowance for loan losses	(4,335)	(5,457)
Acquired loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	934,265	978,346
Acquired loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	14,485	7,729
Acquired loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	41,551	60,034
Originated SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	385,084	387,389
Current fiscal year	6,599	47,939
Year before current fiscal year	47,218	98,568
Two years before current fiscal year	97,040	133,812
Three years before current fiscal year	129,904	68,375
Four years before current fiscal year	66,195	22,056
Five or more years before current fiscal year	25,332	4,041
Total	372,288	374,791
General allowance for loan losses	(4,864)	(5,680)
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	356,600	369,416
Originated SBA 7(a) loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	14,789	1,741
Originated SBA 7(a) loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	899	3,634
Acquired SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	246,778	262,571
Current fiscal year		139
Year before current fiscal year	129	19,658
Two years before current fiscal year	19,485	14,636
Three years before current fiscal year	14,315	283
Four years before current fiscal year	279	19
Five or more years before current fiscal year	190,812	204,703
Total	225,020	239,438
General allowance for loan losses	(627)	(767)
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	221,444	228,651
Acquired SBA 7(a) loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	2,020	4,008
Acquired SBA 7(a) loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	1,556	6,779
Originated SBC loans, at fair value
Loan classification and delinquency by year of origination
UPB	13,881	14,088
Three years before current fiscal year		1,598
Four years before current fiscal year	1,597	6,442
Five or more years before current fiscal year	12,021	5,755
Total	13,618	13,795
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	13,618	13,795
Originated Residential Agency loans
Loan classification and delinquency by year of origination
UPB	3,083	3,208
Current fiscal year	935	1,571
Year before current fiscal year	659	645
Two years before current fiscal year	644	705
Three years before current fiscal year	702
Four years before current fiscal year		88
Five or more years before current fiscal year	198	199
Total	3,138	3,208
Originated Residential Agency loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	936	538
Originated Residential Agency loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total		654
Originated Residential Agency loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 2,202	$ 2,016