Segment Reporting (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2020 USD ($) segment	Mar. 31, 2019 USD ($)
Segment reporting
Number of Reportable Segments | segment	4
Interest income	$ 69,551	$ 48,753
Interest expense	(46,930)	(35,775)
Net interest income before provision for loan losses	22,621	12,978
Provision for loan losses	(39,804)	(518)
Net interest income after provision for loan losses	(17,183)	12,460
Non-interest income
Residential mortgage banking activities	36,669	14,587
Net realized gain (loss) on financial instruments and real estate owned	7,172	7,282
Net unrealized gain (loss) on financial instruments	(33,434)	(6,912)
Servicing income	8,097	6,752
Income on purchased future receivables, net of allowance for doubtful accounts	3,483
Income (Loss) from unconsolidated joint ventures	(3,537)	2,929
Other income	4,073	900
Gain on bargain purchase		30,728
Total non-interest income	22,523	56,266
Non-interest expense
Employee compensation and benefits	(18,936)	(11,448)
Allocated employee compensation and benefits from related party	(1,250)	(853)
Variable expenses on residential mortgage banking activities	(20,129)	(9,176)
Professional fees	(2,556)	(1,829)
Management fees – related party	(2,561)	(1,997)
Loan servicing expense	(5,570)	(3,648)
Merger related expenses	(47)	(5,467)
Other operating expenses	(13,744)	(6,861)
Total non-interest expense	(64,793)	(41,279)
Net income (loss) before provision for income taxes	(59,453)	27,447
Total Assets	5,270,050	3,279,014
Corporate- Other
Non-interest income
Other income	99	15
Gain on bargain purchase		30,728
Total non-interest income	99	30,743
Non-interest expense
Employee compensation and benefits	(742)	(824)
Allocated employee compensation and benefits from related party	(1,125)	(768)
Professional fees	(1,407)	(935)
Management fees – related party	(2,561)	(1,997)
Loan servicing expense	(32)	(35)
Merger related expenses	(47)	(5,467)
Other operating expenses	(698)	(1,200)
Total non-interest expense	(6,612)	(11,226)
Net income (loss) before provision for income taxes	(6,513)	19,517
Total Assets	234,380	61,462
Loan Acquisitions | Operating Segments
Segment reporting
Interest income	16,494	10,674
Interest expense	(11,205)	(7,705)
Net interest income before provision for loan losses	5,289	2,969
Provision for loan losses	(5,722)	(133)
Net interest income after provision for loan losses	(433)	2,836
Non-interest income
Net realized gain (loss) on financial instruments and real estate owned	(739)	(49)
Net unrealized gain (loss) on financial instruments	(9,423)	(19)
Servicing income	355	1
Income on purchased future receivables, net of allowance for doubtful accounts	3,483
Income (Loss) from unconsolidated joint ventures	(3,537)	2,929
Other income	2,336	68
Total non-interest income	(7,525)	2,930
Non-interest expense
Employee compensation and benefits	(2,833)	(1)
Allocated employee compensation and benefits from related party	(125)	(85)
Professional fees	(235)	(170)
Loan servicing expense	(1,365)	(801)
Other operating expenses	(6,245)	(421)
Total non-interest expense	(10,803)	(1,478)
Net income (loss) before provision for income taxes	(18,761)	4,288
Total Assets	1,209,617	864,881
SBC Originations | Operating Segments
Segment reporting
Interest income	39,269	28,188
Interest expense	(25,627)	(20,666)
Net interest income before provision for loan losses	13,642	7,522
Provision for loan losses	(29,828)	41
Net interest income after provision for loan losses	(16,186)	7,563
Non-interest income
Net realized gain (loss) on financial instruments and real estate owned	3,649	3,540
Net unrealized gain (loss) on financial instruments	(6,491)	810
Servicing income	532	436
Other income	1,283	786
Total non-interest income	(1,027)	5,572
Non-interest expense
Employee compensation and benefits	(2,710)	(2,260)
Professional fees	(338)	(301)
Loan servicing expense	(1,580)	(1,215)
Other operating expenses	(3,457)	(2,117)
Total non-interest expense	(8,085)	(5,893)
Net income (loss) before provision for income taxes	(25,298)	7,242
Total Assets	2,704,301	1,823,166
SBA Originations, Acquisitions, and Servicing | Operating Segments
Segment reporting
Interest income	12,471	9,109
Interest expense	(8,513)	(6,490)
Net interest income before provision for loan losses	3,958	2,619
Provision for loan losses	(4,254)	(426)
Net interest income after provision for loan losses	(296)	2,193
Non-interest income
Net realized gain (loss) on financial instruments and real estate owned	4,262	3,791
Net unrealized gain (loss) on financial instruments	(1,082)	(575)
Servicing income	1,074	1,015
Other income	295	14
Total non-interest income	4,549	4,245
Non-interest expense
Employee compensation and benefits	(3,910)	(3,768)
Professional fees	(289)	(187)
Loan servicing expense	(335)	143
Other operating expenses	(1,559)	(1,050)
Total non-interest expense	(6,093)	(4,862)
Net income (loss) before provision for income taxes	(1,840)	1,576
Total Assets	703,331	251,778
Residential Mortgage Banking | Operating Segments
Segment reporting
Interest income	1,317	782
Interest expense	(1,585)	(914)
Net interest income before provision for loan losses	(268)	(132)
Net interest income after provision for loan losses	(268)	(132)
Non-interest income
Residential mortgage banking activities	36,669	14,587
Net unrealized gain (loss) on financial instruments	(16,438)	(7,128)
Servicing income	6,136	5,300
Other income	60	17
Total non-interest income	26,427	12,776
Non-interest expense
Employee compensation and benefits	(8,741)	(4,595)
Variable expenses on residential mortgage banking activities	(20,129)	(9,176)
Professional fees	(287)	(236)
Loan servicing expense	(2,258)	(1,740)
Other operating expenses	(1,785)	(2,073)
Total non-interest expense	(33,200)	(17,820)
Net income (loss) before provision for income taxes	(7,041)	(5,176)
Total Assets	$ 418,421	$ 277,727