Loans and Allowance for Loan Losses - Delinquency (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2020	Dec. 31, 2019	Dec. 31, 2020
Loan delinquency information
Total Loans Carrying Value	$ 4,210,665	$ 4,056,607
Non-Accrual Loans	70,419	84,340
90+ Days Past Due but Accruing	7,419	4,782
General allowance for loan losses	(46,436)	(2,424)	$ (46,436)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 4,164,229	$ 4,054,183	$ 4,164,229
Percentage of outstanding	100.00%	100.00%
Percentage of outstanding, 90+Days Past Due Accruing	0.20%	0.10%
Percentage of outstanding, Non-Accrual Loans	1.70%	2.10%
Specific allowance for loan losses	$ 11,500
Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	$ 4,054,834	$ 3,932,421
Percentage of outstanding	96.30%	97.00%
30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 92,336	$ 77,811
Percentage of outstanding	2.20%	1.90%
90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 63,495	$ 46,375
Percentage of outstanding	1.50%	1.10%
Originated Transitional loans
Loan delinquency information
Total Loans Carrying Value	$ 1,260,761	$ 1,086,328
Non-Accrual Loans	9,890	24,587
Originated Transitional loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,228,782	1,074,955
Originated Transitional loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	22,089	5,728
Originated Transitional loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	9,890	5,645
Originated SBC loans
Loan delinquency information
Total Loans Carrying Value	1,212,817	1,168,899
Non-Accrual Loans	17,568	16,089
Originated SBC loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,184,204	1,137,140
Originated SBC loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	11,045	15,670
Originated SBC loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	17,568	16,089
Acquired loans
Loan delinquency information
Total Loans Carrying Value	1,061,651	1,094,283
Non-Accrual Loans	24,375	23,500
90+ Days Past Due but Accruing	608	3,382
Acquired loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	999,330	1,032,259
Acquired loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	40,759	45,894
Acquired loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	21,562	16,130
Acquired SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	284,078	306,860
Non-Accrual Loans	9,527	9,177
90+ Days Past Due but Accruing	1,315	1,326
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	267,249	297,172
Acquired SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	11,606	4,646
Acquired SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	5,223	5,042
Originated SBC loans, at fair value
Loan delinquency information
Total Loans Carrying Value	19,813	20,212
90+ Days Past Due but Accruing	5,496
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	14,317	20,212
Originated SBC loans, at fair value | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	5,496
Originated SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	367,429	376,629
Non-Accrual Loans	6,484	8,882
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	359,411	370,101
Originated SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	6,257	5,238
Originated SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	1,761	1,290
Originated Residential Agency loans
Loan delinquency information
Total Loans Carrying Value	4,116	3,396
Non-Accrual Loans	2,575	2,105
90+ Days Past Due but Accruing		74
Originated Residential Agency loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,541	582
Originated Residential Agency loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	580	635
Originated Residential Agency loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 1,995	$ 2,179