Interest Income and Interest Expense (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017
Interest income
Total loans	$ 39,456	$ 30,711	$ 74,322	$ 62,436
Total loans, held for sale, at fair value	1,495	1,638	2,896	3,072
Mortgage backed securities, at fair value	907	899	1,789	1,622
Total interest income	41,858	33,248	79,007	67,130
Interest expense
Secured borrowings	(8,708)	(7,106)	(15,890)	(14,304)
Securitized debt obligations of consolidated VIEs	(8,288)	(4,761)	(15,491)	(9,884)
Guaranteed loan financing	(2,956)	(3,520)	(5,770)	(6,784)
Senior secured note	(3,487)	(1,777)	(6,726)	(2,567)
Convertible note	(2,187)		(4,373)
Corporate debt	(742)		(742)
Promissory note	(39)	(66)	(79)	(134)
Total interest expense	(26,407)	(17,230)	(49,071)	(33,673)
Net interest income before provision for loan losses	15,451	16,018	29,936	33,457
Acquired SBA 7(a) loans
Interest income
Total loans	7,968	9,663	15,840	19,540
Acquired loans
Interest income
Total loans	12,707	8,685	22,166	18,728
Total loans, held for sale, at fair value	77	372	90	793
Originated Transitional loans
Interest income
Total loans	10,038	3,945	18,950	8,048
Originated SBC loans, at fair value
Interest income
Total loans	497	2,204	2,525	3,830
Originated SBC loans
Interest income
Total loans	7,204	6,100	12,944	11,711
Originated SBA 7(a) loans
Interest income
Total loans	1,030	89	1,873	354
Originated Residential Agency loans
Interest income
Total loans	12	25	24	225
Total loans, held for sale, at fair value	987	1,025	1,864	1,855
Originated Freddie Mac loans
Interest income
Total loans, held for sale, at fair value	$ 431	$ 241	$ 942	$ 424