Loans and Allowance for Loan Losses - Loan Vintage (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Loan classification and delinquency by year of origination
UPB	$ 4,176,966
2020	597,892
2019	1,229,069
2018	735,604
2017	243,042
2016	110,314
Pre 2016	1,211,980
Total	4,127,901	$ 4,056,607
General allowance for loan losses	(29,539)	(2,424)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	4,098,362	4,054,183
Specific allowance for loan losses	17,200	5,000
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	3,979,225
2020	591,405
2019	1,221,227
2018	707,068
2017	203,331
2016	100,003
Pre 2016	1,125,100
Total	3,948,134	3,932,421
30-59 Days Past Due
Loan classification and delinquency by year of origination
UPB	38,836
2020	5,812
2019	5,191
2018	15,097
2017	401
2016	2
Pre 2016	11,933
Total	38,436	65,669
60+ Days Past Due
Loan classification and delinquency by year of origination
UPB	158,905
2020	675
2019	2,651
2018	13,439
2017	39,310
2016	10,309
Pre 2016	74,947
Total	141,331	58,517
Originated Transitional loans
Loan classification and delinquency by year of origination
UPB	1,328,395
2020	385,183
2019	583,593
2018	306,971
2017	23,783
2016	18,480
Pre 2016	1,064
Total	1,319,074	1,086,328
General allowance for loan losses	(14,995)	(64)
Originated Transitional loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,281,579	1,074,955
Originated Transitional loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	17,713	5,728
Originated Transitional loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	19,782	5,645
Originated SBC loans
Loan classification and delinquency by year of origination
UPB	1,052,705
2020	66,715
2019	486,033
2018	237,313
2017	110,354
2016	43,696
Pre 2016	112,444
Total	1,056,555	1,168,899
General allowance for loan losses	(2,640)	(124)
Originated SBC loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,000,878	1,137,140
Originated SBC loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	6,591	11,769
Originated SBC loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	49,086	19,990
Acquired loans
Loan classification and delinquency by year of origination
UPB	1,053,679
2020	21,414
2019	40,572
2018	42,167
2017	38,649
2016	19,533
Pre 2016	883,774
Total	1,046,109	1,094,283
General allowance for loan losses	(5,457)	(803)
Acquired loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	978,346	1,032,259
Acquired loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	7,729	41,830
Acquired loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	60,034	20,194
Originated SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	387,389
2020	47,939
2019	98,568
2018	133,812
2017	68,375
2016	22,056
Pre 2016	4,041
Total	374,791	376,629
General allowance for loan losses	(5,680)	(1,019)
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	369,416	370,101
Originated SBA 7(a) loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	1,741	2,085
Originated SBA 7(a) loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	3,634	4,443
Acquired SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	262,571
2020	139
2019	19,658
2018	14,636
2017	283
2016	19
Pre 2016	204,703
Total	239,438	306,860
General allowance for loan losses	(767)	(414)
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	228,651	297,172
Acquired SBA 7(a) loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	4,008	4,048
Acquired SBA 7(a) loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	6,779	5,640
Originated PPP loans, at fair value
Loan classification and delinquency by year of origination
UPB	74,931
2020	74,931
Total	74,931
Originated PPP loans, at fair value | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	74,931
Originated SBC loans, at fair value
Loan classification and delinquency by year of origination
UPB	14,088
2017	1,598
2016	6,442
Pre 2016	5,755
Total	13,795	20,212
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	13,795	20,212
Originated Residential Agency loans
Loan classification and delinquency by year of origination
UPB	3,208
2020	1,571
2019	645
2018	705
2016	88
Pre 2016	199
Total	3,208	3,396
Originated Residential Agency loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	538	582
Originated Residential Agency loans | 30-59 Days Past Due
Loan classification and delinquency by year of origination
Total	654	209
Originated Residential Agency loans | 60+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 2,016	$ 2,605