Segment Reporting (Details) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019 USD ($)	Sep. 30, 2018 USD ($)	Sep. 30, 2019 USD ($) segment	Sep. 30, 2018 USD ($)
Segment reporting
Number of Reportable Segments | segment			4
Interest income	$ 59,723	$ 44,287	$ 165,510	$ 123,295
Interest expense	(39,390)	(28,925)	(110,919)	(77,996)
Net interest income before provision for loan losses	20,333	15,362	54,591	45,299
Provision for loan losses	(693)	(800)	(2,559)	(571)
Net interest income after provision for loan losses	19,640	14,562	52,032	44,728
Non-interest income
Residential mortgage banking activities	29,013	17,011	64,621	48,290
Net realized gain on financial instruments	7,377	6,946	20,914	27,796
Net unrealized gain (loss) on financial instruments	(7,881)	8,500	(21,799)	15,964
Other income	2,979	1,204	6,671	4,364
Servicing income	7,449	6,922	22,012	19,959
Income on unconsolidated joint ventures	1,047	2,178	6,059	9,420
Gain on bargain purchase			30,728
Total non-interest income	39,984	42,761	129,206	125,793
Non-interest expense
Employee compensation and benefits	(13,438)	(14,163)	(37,395)	(43,755)
Allocated employee compensation and benefits from related party	(1,500)	(1,200)	(3,603)	(3,600)
Variable expenses on residential mortgage banking activities	(17,318)	(8,337)	(39,995)	(18,120)
Professional fees	(2,030)	(2,294)	(5,445)	(7,343)
Management fees – related party	(2,495)	(2,070)	(6,987)	(6,118)
Incentive fees – related party				(676)
Loan servicing expense	(4,866)	(4,247)	(13,085)	(11,340)
Merger related expenses	(51)		(6,121)
Other operating expenses	(8,144)	(6,548)	(23,091)	(23,475)
Total non-interest expense	(49,842)	(38,859)	(135,722)	(114,427)
Net income (loss) before provision for income taxes	9,782	18,464	45,516	56,094
Total Assets	4,123,019	2,900,759	4,123,019	2,900,759
Corporate- Other
Non-interest income
Other income	44		109
Gain on bargain purchase			30,728
Total non-interest income	44		30,837
Non-interest expense
Employee compensation and benefits	(831)	(157)	(2,495)	(471)
Allocated employee compensation and benefits from related party	(1,350)	(1,080)	(3,243)	(3,240)
Professional fees	(641)	(1,001)	(2,404)	(3,642)
Management fees – related party	(2,495)	(2,070)	(6,987)	(6,118)
Incentive fees – related party				(676)
Loan servicing expense	(33)		(110)
Merger related expenses	(51)		(6,121)
Other operating expenses	(806)	(544)	(2,367)	(1,927)
Total non-interest expense	(6,207)	(4,852)	(23,727)	(16,074)
Net income (loss) before provision for income taxes	(6,163)	(4,852)	7,110	(16,074)
Total Assets	91,094	24,918	91,094	24,918
Loan Acquisitions
Segment reporting
Interest income	18,880	11,946	47,761	34,602
Interest expense	(11,076)	(7,755)	(29,383)	(19,853)
Net interest income before provision for loan losses	7,804	4,191	18,378	14,749
Provision for loan losses	(94)	(796)	(771)	(989)
Net interest income after provision for loan losses	7,710	3,395	17,607	13,760
Non-interest income
Net realized gain on financial instruments	(437)	774	(33)	867
Net unrealized gain (loss) on financial instruments	(206)	2,893	(886)	3,695
Other income	925	75	1,950	342
Servicing income		5		15
Income on unconsolidated joint ventures	1,047	2,178	6,059	9,420
Total non-interest income	1,329	5,925	7,090	14,339
Non-interest expense
Employee compensation and benefits	(26)	(98)	(53)	(410)
Allocated employee compensation and benefits from related party	(150)	(120)	(360)	(360)
Professional fees	(195)	(467)	(412)	(928)
Loan servicing expense	(1,461)	(1,183)	(3,379)	(2,748)
Other operating expenses	(770)	(450)	(2,073)	(3,267)
Total non-interest expense	(2,602)	(2,318)	(6,277)	(7,713)
Net income (loss) before provision for income taxes	6,437	7,002	18,420	20,386
Total Assets	1,165,470	672,569	1,165,470	672,569
SBC Originations
Segment reporting
Interest income	32,354	21,671	91,182	58,423
Interest expense	(23,835)	(16,045)	(65,903)	(43,838)
Net interest income before provision for loan losses	8,519	5,626	25,279	14,585
Provision for loan losses	(33)	(101)	(342)	8
Net interest income after provision for loan losses	8,486	5,525	24,937	14,593
Non-interest income
Net realized gain on financial instruments	5,065	3,253	10,232	15,484
Net unrealized gain (loss) on financial instruments	(865)	2,679	(131)	4,441
Other income	1,118	794	3,565	3,409
Servicing income	488	389	1,386	972
Total non-interest income	5,806	7,115	15,052	24,306
Non-interest expense
Employee compensation and benefits	(1,420)	(2,417)	(5,454)	(7,339)
Professional fees	(571)	(291)	(1,185)	(1,012)
Loan servicing expense	(1,495)	(1,294)	(4,185)	(2,819)
Other operating expenses	(3,369)	(2,545)	(8,198)	(8,512)
Total non-interest expense	(6,855)	(6,547)	(19,022)	(19,682)
Net income (loss) before provision for income taxes	7,437	6,093	20,967	19,217
Total Assets	2,266,076	1,442,601	2,266,076	1,442,601
SBA Originations, Acquisitions, and Servicing
Segment reporting
Interest income	7,235	9,699	23,468	27,411
Interest expense	(2,739)	(4,301)	(11,529)	(11,892)
Net interest income before provision for loan losses	4,496	5,398	11,939	15,519
Provision for loan losses	(566)	97	(1,446)	410
Net interest income after provision for loan losses	3,930	5,495	10,493	15,929
Non-interest income
Net realized gain on financial instruments	2,749	2,919	10,715	11,445
Net unrealized gain (loss) on financial instruments	772	611	268	325
Other income	812	279	865	447
Servicing income	1,180	1,352	4,013	4,067
Total non-interest income	5,513	5,161	15,861	16,284
Non-interest expense
Employee compensation and benefits	(4,070)	(3,116)	(12,262)	(9,510)
Professional fees	(279)	(412)	(635)	(1,414)
Loan servicing expense	(205)	(198)	(322)	89
Other operating expenses	(1,508)	(929)	(4,635)	(3,009)
Total non-interest expense	(6,062)	(4,655)	(17,854)	(13,844)
Net income (loss) before provision for income taxes	3,381	6,001	8,500	18,369
Total Assets	273,217	484,387	273,217	484,387
Residential Mortgage Banking
Segment reporting
Interest income	1,254	971	3,099	2,859
Interest expense	(1,740)	(824)	(4,104)	(2,413)
Net interest income before provision for loan losses	(486)	147	(1,005)	446
Net interest income after provision for loan losses	(486)	147	(1,005)	446
Non-interest income
Residential mortgage banking activities	29,013	17,011	64,621	48,290
Net unrealized gain (loss) on financial instruments	(7,582)	2,317	(21,050)	7,503
Other income	80	56	182	166
Servicing income	5,781	5,176	16,613	14,905
Total non-interest income	27,292	24,560	60,366	70,864
Non-interest expense
Employee compensation and benefits	(7,091)	(8,375)	(17,131)	(26,025)
Variable expenses on residential mortgage banking activities	(17,318)	(8,337)	(39,995)	(18,120)
Professional fees	(344)	(123)	(809)	(347)
Loan servicing expense	(1,672)	(1,572)	(5,089)	(5,862)
Other operating expenses	(1,691)	(2,080)	(5,818)	(6,760)
Total non-interest expense	(28,116)	(20,487)	(68,842)	(57,114)
Net income (loss) before provision for income taxes	(1,310)	4,220	(9,481)	14,196
Total Assets	$ 327,162	$ 276,284	$ 327,162	$ 276,284