Loans and Allowance for Loan Losses - Delinquency (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2020	Dec. 31, 2019
Loan delinquency information
Total Loans Carrying Value	$ 4,197,950	$ 4,056,607
Non-Accrual Loans	88,086	84,340
90+ Days Past Due but Accruing	12,659	4,782
General allowance for loan losses	(42,969)	(2,424)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 4,154,981	$ 4,054,183
Percentage of outstanding	100.00%	100.00%
Percentage of outstanding, 90+Days Past Due Accruing	0.30%	0.10%
Percentage of outstanding, Non-Accrual Loans	2.10%	2.10%
Specific allowance for loan losses	$ 14,100	$ 4,000
Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	$ 3,810,725	$ 3,932,421
Percentage of outstanding	90.70%	97.00%
30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 305,073	$ 77,811
Percentage of outstanding	7.30%	1.90%
90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 82,152	$ 46,375
Percentage of outstanding	2.00%	1.10%
Originated Transitional loans
Loan delinquency information
Total Loans Carrying Value	$ 1,268,775	$ 1,086,328
Non-Accrual Loans	24,664	24,587
Originated Transitional loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,186,742	1,074,955
Originated Transitional loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	57,369	5,728
Originated Transitional loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	24,664	5,645
Originated SBC loans
Loan delinquency information
Total Loans Carrying Value	1,174,414	1,168,899
Non-Accrual Loans	13,810	16,089
Originated SBC loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,093,804	1,137,140
Originated SBC loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	66,800	15,670
Originated SBC loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	13,810	16,089
Acquired loans
Loan delinquency information
Total Loans Carrying Value	997,830	1,094,283
Non-Accrual Loans	31,087	23,500
90+ Days Past Due but Accruing	57	3,382
Acquired loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	793,243	1,032,259
Acquired loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	179,745	45,894
Acquired loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	24,842	16,130
Acquired SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	266,138	306,860
Non-Accrual Loans	9,770	9,177
90+ Days Past Due but Accruing	1,300	1,326
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	261,189	297,172
Acquired SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	630	4,646
Acquired SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	4,319	5,042
Originated SBC loans, at fair value
Loan delinquency information
Total Loans Carrying Value	19,050	20,212
90+ Days Past Due but Accruing	10,040
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	9,011	20,212
Originated SBC loans, at fair value | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	10,039
Originated SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	362,200	376,629
Non-Accrual Loans	6,372	8,882
90+ Days Past Due but Accruing	837
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	360,355	370,101
Originated SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value		5,238
Originated SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	1,845	1,290
Originated Residential Agency loans
Loan delinquency information
Total Loans Carrying Value	4,295	3,396
Non-Accrual Loans	2,383	2,105
90+ Days Past Due but Accruing	425	74
Originated Residential Agency loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,133	582
Originated Residential Agency loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	529	635
Originated Residential Agency loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 2,633	$ 2,179