Segment Reporting (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020 USD ($)	Jun. 30, 2019 USD ($)	Jun. 30, 2020 USD ($) segment	Jun. 30, 2019 USD ($)
Segment reporting
Number of Reportable Segments | segment			4
Interest income	$ 63,211	$ 57,034	$ 132,762	$ 105,787
Interest expense	(43,408)	(35,753)	(90,338)	(71,529)
Net interest income before provision for loan losses	19,803	21,281	42,424	34,258
Provision for loan losses	591	(1,348)	(39,214)	(1,866)
Net interest income after provision for loan losses	20,394	19,933	3,210	32,392
Non-interest income
Residential mortgage banking activities	80,564	21,021	117,233	35,608
Net realized gains on financial instruments and real estate owned	7,438	6,255	14,610	13,537
Net unrealized gain (loss) on financial instruments	(13,744)	(7,006)	(47,178)	(13,918)
Servicing income	8,982	7,811	17,079	14,563
Income on purchased future receivables, net of allowance for doubtful accounts	5,586		9,069
Income (Loss) from unconsolidated joint ventures	507	2,083	(3,030)	5,012
Other income	31,594	2,792	35,667	3,692
Gain on bargain purchase				30,728
Total non-interest income (loss)	120,927	32,956	143,450	89,222
Non-interest expense
Employee compensation and benefits	(27,288)	(12,509)	(46,224)	(23,957)
Allocated employee compensation and benefits from related party	(1,250)	(1,250)	(2,500)	(2,103)
Variable expenses on residential mortgage banking activities	(36,446)	(13,501)	(56,575)	(22,677)
Professional fees	(1,919)	(1,586)	(4,475)	(3,415)
Management fees - related party	(2,666)	(2,495)	(5,227)	(4,492)
Incentive fees - related party	(3,506)		(3,506)
Loan servicing expense	(10,327)	(4,571)	(15,898)	(8,219)
Merger related expenses	(11)	(603)	(58)	(6,070)
Other operating expenses	(17,745)	(8,085)	(31,487)	(14,947)
Total non-interest expense	(101,158)	(44,600)	(165,950)	(85,880)
Net income (loss) before provision for income taxes	40,163	8,289	(19,290)	35,734
Total Assets	5,460,932	3,840,323	5,460,932	3,840,323
Corporate- Other
Segment reporting
Interest expense	(372)		(372)
Net interest income before provision for loan losses	(372)		(372)
Net interest income after provision for loan losses	(372)		(372)
Non-interest income
Other income	16	50	115	65
Gain on bargain purchase				30,728
Total non-interest income (loss)	16	50	115	30,793
Non-interest expense
Employee compensation and benefits	(633)	(840)	(1,375)	(1,664)
Allocated employee compensation and benefits from related party	(1,125)	(1,125)	(2,250)	(1,893)
Professional fees	(1,155)	(828)	(2,562)	(1,763)
Management fees - related party	(2,666)	(2,495)	(5,227)	(4,492)
Incentive fees - related party	(3,506)		(3,506)
Loan servicing expense	(8)	(42)	(40)	(77)
Merger related expenses	(11)	(603)	(58)	(6,070)
Other operating expenses	(741)	(361)	(1,437)	(1,561)
Total non-interest expense	(9,845)	(6,294)	(16,455)	(17,520)
Net income (loss) before provision for income taxes	(10,201)	(6,244)	(16,712)	13,273
Total Assets	342,783	61,430	342,783	61,430
Loan Acquisitions | Operating Segments
Segment reporting
Interest income	14,977	18,208	31,470	28,881
Interest expense	(10,654)	(10,602)	(21,859)	(18,307)
Net interest income before provision for loan losses	4,323	7,606	9,611	10,574
Provision for loan losses	(1,965)	(544)	(7,688)	(677)
Net interest income after provision for loan losses	2,358	7,062	1,923	9,897
Non-interest income
Net realized gains on financial instruments and real estate owned	(396)	453	(1,135)	404
Net unrealized gain (loss) on financial instruments	(1,016)	(661)	(10,439)	(680)
Servicing income	172		527	1
Income on purchased future receivables, net of allowance for doubtful accounts	5,586		9,069
Income (Loss) from unconsolidated joint ventures	507	2,083	(3,030)	5,012
Other income	1,419	957	3,755	1,025
Total non-interest income (loss)	6,272	2,832	(1,253)	5,762
Non-interest expense
Employee compensation and benefits	(2,638)	(26)	(5,471)	(27)
Allocated employee compensation and benefits from related party	(125)	(125)	(250)	(210)
Professional fees	(251)	(47)	(486)	(217)
Loan servicing expense	(1,500)	(1,117)	(2,866)	(1,918)
Other operating expenses	(4,072)	(882)	(10,317)	(1,304)
Total non-interest expense	(8,586)	(2,197)	(19,390)	(3,676)
Net income (loss) before provision for income taxes	44	7,697	(18,720)	11,983
Total Assets	1,125,035	1,158,351	1,125,035	1,158,351
SBC Originations | Operating Segments
Segment reporting
Interest income	37,497	30,639	76,766	58,828
Interest expense	(23,507)	(21,401)	(49,134)	(42,068)
Net interest income before provision for loan losses	13,990	9,238	27,632	16,760
Provision for loan losses	5,821	(350)	(24,007)	(309)
Net interest income after provision for loan losses	19,811	8,888	3,625	16,451
Non-interest income
Net realized gains on financial instruments and real estate owned	6,232	1,627	9,881	5,167
Net unrealized gain (loss) on financial instruments	(716)	(76)	(7,207)	734
Servicing income	399	461	931	897
Other income	1,439	1,661	2,722	2,447
Total non-interest income (loss)	7,354	3,673	6,327	9,245
Non-interest expense
Employee compensation and benefits	(4,689)	(1,774)	(7,399)	(4,034)
Professional fees	(104)	(313)	(442)	(614)
Loan servicing expense	(1,711)	(1,475)	(3,291)	(2,690)
Other operating expenses	(4,429)	(2,712)	(7,886)	(4,829)
Total non-interest expense	(10,933)	(6,274)	(19,018)	(12,167)
Net income (loss) before provision for income taxes	16,232	6,287	(9,066)	13,529
Total Assets	2,620,406	2,055,751	2,620,406	2,055,751
SBA Originations, Acquisitions, and Servicing | Operating Segments
Segment reporting
Interest income	8,808	7,124	21,279	16,233
Interest expense	(6,839)	(2,300)	(15,352)	(8,790)
Net interest income before provision for loan losses	1,969	4,824	5,927	7,443
Provision for loan losses	(2,765)	(454)	(7,019)	(880)
Net interest income after provision for loan losses	(796)	4,370	(1,092)	6,563
Non-interest income
Net realized gains on financial instruments and real estate owned	1,602	4,175	5,864	7,966
Net unrealized gain (loss) on financial instruments	31	71	(1,051)	(504)
Servicing income	2,393	1,818	3,467	2,833
Other income	28,674	39	28,969	53
Total non-interest income (loss)	32,700	6,103	37,249	10,348
Non-interest expense
Employee compensation and benefits	(3,485)	(4,424)	(7,395)	(8,192)
Professional fees	(138)	(169)	(427)	(356)
Loan servicing expense	(247)	(260)	(582)	(117)
Other operating expenses	(6,530)	(2,076)	(8,089)	(3,126)
Total non-interest expense	(10,400)	(6,929)	(16,493)	(11,791)
Net income (loss) before provision for income taxes	21,504	3,544	19,664	5,120
Total Assets	804,355	261,326	804,355	261,326
Residential Mortgage Banking | Operating Segments
Segment reporting
Interest income	1,929	1,063	3,247	1,845
Interest expense	(2,036)	(1,450)	(3,621)	(2,364)
Net interest income before provision for loan losses	(107)	(387)	(374)	(519)
Provision for loan losses	(500)		(500)
Net interest income after provision for loan losses	(607)	(387)	(874)	(519)
Non-interest income
Residential mortgage banking activities	80,564	21,021	117,233	35,608
Net unrealized gain (loss) on financial instruments	(12,043)	(6,340)	(28,481)	(13,468)
Servicing income	6,018	5,532	12,154	10,832
Other income	46	85	106	102
Total non-interest income (loss)	74,585	20,298	101,012	33,074
Non-interest expense
Employee compensation and benefits	(15,843)	(5,445)	(24,584)	(10,040)
Variable expenses on residential mortgage banking activities	(36,446)	(13,501)	(56,575)	(22,677)
Professional fees	(271)	(229)	(558)	(465)
Loan servicing expense	(6,861)	(1,677)	(9,119)	(3,417)
Other operating expenses	(1,973)	(2,054)	(3,758)	(4,127)
Total non-interest expense	(61,394)	(22,906)	(94,594)	(40,726)
Net income (loss) before provision for income taxes	12,584	(2,995)	5,544	(8,171)
Total Assets	$ 568,353	$ 303,465	$ 568,353	$ 303,465