Loans and Allowance for Loan Losses - Delinquency (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2019	Dec. 31, 2018
Loan delinquency information
Total Loans Carrying Value	$ 3,311,156	$ 2,416,303
Non-Accrual Loans	71,319	49,486
90+ Days Past Due but Accruing	3,160	4,854
General allowance for loan losses	(2,264)	(1,937)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 3,308,892	$ 2,414,366
Percentage of outstanding	100.00%	100.00%
Financing Receivable, Nonaccrual, Percent Past Due	2.20%	2.00%
Percentage of outstanding	0.10%	0.20%
Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	$ 3,226,102	$ 2,337,336
Percentage of outstanding	97.40%	96.70%
30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 41,849	$ 42,331
Percentage of outstanding	1.30%	1.80%
90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 43,205	$ 36,636
Percentage of outstanding	1.30%	1.50%
Acquired SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	$ 106,668	$ 318,488
Non-Accrual Loans	10,269	17,916
90+ Days Past Due but Accruing	1,314	1,043
General allowance for loan losses	(347)	(532)
Acquired SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	99,868	299,080
Acquired SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	1,212	14,943
Acquired SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	5,588	4,465
Acquired loans
Loan delinquency information
Total Loans Carrying Value	967,995	545,695
Non-Accrual Loans	20,435	11,447
90+ Days Past Due but Accruing	1,846	3,811
General allowance for loan losses	(780)	(608)
Acquired loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	927,144	524,930
Acquired loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	25,441	7,213
Acquired loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	15,410	13,552
Acquired Transitional loans
Loan delinquency information
Total Loans Carrying Value	83,355
Non-Accrual Loans	4,886
General allowance for loan losses	(62)
Acquired Transitional loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	78,325
Acquired Transitional loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	2,716
Acquired Transitional loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	2,314
Originated Transitional loans
Loan delinquency information
Total Loans Carrying Value	899,179	664,733
Non-Accrual Loans	18,094
General allowance for loan losses	(290)	(353)
Originated Transitional loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	889,110	659,103
Originated Transitional loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	4,424	5,630
Originated Transitional loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	5,645
Originated SBC loans, at fair value
Loan delinquency information
Total Loans Carrying Value	20,434	22,664
Originated SBC loans, at fair value | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	20,434	22,664
Originated SBC loans
Loan delinquency information
Total Loans Carrying Value	1,119,872	777,408
Non-Accrual Loans	10,788	16,895
General allowance for loan losses	(23)	(11)
Originated SBC loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	1,101,103	748,146
Originated SBC loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	7,981	12,367
Originated SBC loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	10,788	16,895
Originated SBA 7(a) loans
Loan delinquency information
Total Loans Carrying Value	110,784	85,416
Non-Accrual Loans	4,511	1,666
General allowance for loan losses	(762)	(433)
Originated SBA 7(a) loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	109,585	83,076
Originated SBA 7(a) loans | 30-89 Days Past Due
Loan delinquency information
Total Loans Carrying Value	75	2,178
Originated SBA 7(a) loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	1,124	162
Originated Residential Agency loans
Loan delinquency information
Total Loans Carrying Value	2,869	1,899
Non-Accrual Loans	2,336	1,562
Originated Residential Agency loans | Current and less than 30 days past due
Loan delinquency information
Total Loans Carrying Value	533	337
Originated Residential Agency loans | 90+ Days Past Due
Loan delinquency information
Total Loans Carrying Value	$ 2,336	$ 1,562