Loans and Allowance for Loan Losses - Loan Vintage (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Loan classification and delinquency by year of origination
UPB	$ 4,245,329
2020	392,241
2019	1,342,593
2018	792,537
2017	278,471
2016	122,827
Pre 2016	1,269,281
Total	4,197,950	$ 4,056,607
General allowance for loan losses	(42,969)	(2,424)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	4,154,981	4,054,183
Specific allowance for loan losses	14,100	4,000
Current and less than 30 days past due
Loan classification and delinquency by year of origination
UPB	3,846,470
2020	391,330
2019	1,291,375
2018	737,627
2017	231,941
2016	92,100
Pre 2016	1,066,352
Total	3,810,725	3,932,421
30-89 Days Past Due
Loan classification and delinquency by year of origination
UPB	303,591
2020	529
2019	49,852
2018	40,313
2017	30,283
2016	9,790
Pre 2016	174,306
Total	305,073	77,811
90+ Days Past Due
Loan classification and delinquency by year of origination
UPB	95,268
2020	382
2019	1,366
2018	14,597
2017	16,247
2016	20,937
Pre 2016	28,623
Total	82,152	46,375
Originated Transitional loans
Loan classification and delinquency by year of origination
UPB	1,277,526
2020	196,317
2019	648,447
2018	337,312
2017	58,618
2016	26,990
Pre 2016	1,091
Total	1,268,775	1,086,328
Originated Transitional loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,186,742	1,074,955
Originated Transitional loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total	57,369	5,728
Originated Transitional loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	24,664	5,645
Originated SBC loans
Loan classification and delinquency by year of origination
UPB	1,166,644
2020	72,024
2019	538,488
2018	273,034
2017	113,472
2016	46,030
Pre 2016	131,366
Total	1,174,414	1,168,899
Originated SBC loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,093,804	1,137,140
Originated SBC loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total	66,800	15,670
Originated SBC loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	13,810	16,089
Acquired loans
Loan classification and delinquency by year of origination
UPB	1,005,877
2020	2,501
2019	29,867
2018	22,921
2017	28,023
2016	15,554
Pre 2016	898,964
Total	997,830	1,094,283
Acquired loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	793,243	1,032,259
Acquired loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total	179,745	45,894
Acquired loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	24,842	16,130
Acquired SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	293,029
2020	151
2019	23,477
2018	14,900
2017	289
2016	31
Pre 2016	227,290
Total	266,138	306,860
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	261,189	297,172
Acquired SBA 7(a) loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total	630	4,646
Acquired SBA 7(a) loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	4,319	5,042
Originated SBA 7(a) loans
Loan classification and delinquency by year of origination
UPB	373,159
2020	14,712
2019	100,526
2018	143,662
2017	76,160
2016	22,672
Pre 2016	4,468
Total	362,200	376,629
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	360,355	370,101
Originated SBA 7(a) loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total		5,238
Originated SBA 7(a) loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	1,845	1,290
Originated PPP loans, at fair value
Loan classification and delinquency by year of origination
UPB	105,248
2020	105,248
Total	105,248
Originated PPP loans, at fair value | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	105,248
Originated SBC loans, at fair value
Loan classification and delinquency by year of origination
UPB	19,551
2017	1,690
2016	11,460
Pre 2016	5,900
Total	19,050	20,212
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	9,011	20,212
Originated SBC loans, at fair value | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	10,039
Originated Residential Agency loans
Loan classification and delinquency by year of origination
UPB	4,295
2020	1,288
2019	1,788
2018	708
2017	219
2016	90
Pre 2016	202
Total	4,295	3,396
Originated Residential Agency loans | Current and less than 30 days past due
Loan classification and delinquency by year of origination
Total	1,133	582
Originated Residential Agency loans | 30-89 Days Past Due
Loan classification and delinquency by year of origination
Total	529	635
Originated Residential Agency loans | 90+ Days Past Due
Loan classification and delinquency by year of origination
Total	$ 2,633	$ 2,179