Segment Reporting (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2021 USD ($) segment	Dec. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)
Segment reporting
Number of reportable segments | segment	3
Interest income	$ 403,496	$ 258,636	$ 229,916
Interest expense	(213,561)	(175,481)	(151,880)
Net interest income before provision for loan losses	189,935	83,155	78,036
Provision for (recovery of) loan losses	(8,049)	(34,726)	(3,684)
Net interest income after provision for (recovery of) loan losses	181,886	48,429	74,352
Non-interest income
Residential mortgage banking activities	137,297	252,720	83,539
Net realized gains on financial instruments and real estate owned	68,881	31,913	28,958
Net unrealized gain (loss) on financial instruments	39,377	(48,101)	(18,790)
Servicing income, net	48,015	38,594	30,665
Income on purchased future receivables, net	10,257	15,711	2,362
Income (Loss) from unconsolidated joint ventures	6,916	2,404	6,088
Other income (loss)	9,009	41,516	11,078
Gain on bargain purchase			30,728
Total non-interest income (loss)	319,752	334,757	174,628
Non-interest expense
Employee compensation and benefits	(90,065)	(91,920)	(51,237)
Allocated employee compensation and benefits from related party	(12,031)	(7,000)	(5,473)
Variable expenses on residential mortgage banking activities	(75,133)	(114,510)	(51,760)
Professional fees	(16,339)	(13,360)	(7,434)
Management fees - related party	(10,928)	(10,682)	(9,578)
Incentive fees - related party	(5,419)	(5,973)	(106)
Loan servicing expense	(29,983)	(30,856)	(17,976)
Transaction related expenses	(14,282)	(63)	(7,750)
Other operating expenses	(58,401)	(54,369)	(33,162)
Total non-interest expense	(312,581)	(328,733)	(184,476)
Income (loss) before provision for income taxes	189,057	54,453	64,504
Total assets	9,534,031	5,372,095	4,977,018
Operating Segments | SBC Lending and Acquisitions
Segment reporting
Interest income	278,455	211,525	193,417
Interest expense	(164,797)	(138,444)	(131,179)
Net interest income before provision for loan losses	113,658	73,081	62,238
Provision for (recovery of) loan losses	(7,387)	(26,932)	(1,127)
Net interest income after provision for (recovery of) loan losses	106,271	46,149	61,111
Non-interest income
Net realized gains on financial instruments and real estate owned	24,813	13,349	13,768
Net unrealized gain (loss) on financial instruments	19,457	(9,763)	(361)
Servicing income, net	3,113	2,265	1,899
Income (Loss) from unconsolidated joint ventures	6,916	2,404	6,088
Other income (loss)	13,002	8,200	8,304
Total non-interest income (loss)	67,301	16,455	29,698
Non-interest expense
Employee compensation and benefits	(16,582)	(14,137)	(6,958)
Allocated employee compensation and benefits from related party	(1,203)	(700)	(547)
Professional fees	(4,064)	(2,174)	(2,134)
Loan servicing expense	(19,680)	(14,390)	(10,357)
Other operating expenses	(21,997)	(18,865)	(14,218)
Total non-interest expense	(63,526)	(50,266)	(34,214)
Income (loss) before provision for income taxes	110,046	12,338	56,595
Total assets	7,106,206	3,790,899	3,694,266
Operating Segments | Small Business Lending
Segment reporting
Interest income	116,741	39,430	32,096
Interest expense	(36,872)	(27,472)	(14,864)
Net interest income before provision for loan losses	79,869	11,958	17,232
Provision for (recovery of) loan losses	(662)	(7,794)	(2,557)
Net interest income after provision for (recovery of) loan losses	79,207	4,164	14,675
Non-interest income
Net realized gains on financial instruments and real estate owned	44,068	18,564	15,190
Net unrealized gain (loss) on financial instruments	2,999	(1,084)	393
Servicing income, net	14,510	11,100	6,127
Income on purchased future receivables, net	10,257	15,711	2,362
Other income (loss)	(6,231)	32,944	2,361
Total non-interest income (loss)	65,603	77,235	26,433
Non-interest expense
Employee compensation and benefits	(36,757)	(29,008)	(18,057)
Professional fees	(3,034)	(1,839)	(1,252)
Loan servicing expense	(886)	(675)	(261)
Other operating expenses	(23,377)	(23,525)	(8,145)
Total non-interest expense	(64,054)	(55,047)	(27,715)
Income (loss) before provision for income taxes	80,756	26,352	13,393
Total assets	1,558,641	754,600	822,147
Operating Segments | Residential Mortgage Banking
Segment reporting
Interest income	8,300	7,681	4,403
Interest expense	(9,193)	(8,294)	(5,837)
Net interest income before provision for loan losses	(893)	(613)	(1,434)
Net interest income after provision for (recovery of) loan losses	(893)	(613)	(1,434)
Non-interest income
Residential mortgage banking activities	137,297	252,720	83,539
Net unrealized gain (loss) on financial instruments	16,921	(37,254)	(18,569)
Servicing income, net	30,392	25,229	22,639
Other income (loss)	2,153	183	249
Total non-interest income (loss)	186,763	240,878	87,858
Non-interest expense
Employee compensation and benefits	(32,973)	(45,368)	(22,882)
Variable expenses on residential mortgage banking activities	(75,133)	(114,510)	(51,760)
Professional fees	(2,951)	(1,777)	(1,101)
Loan servicing expense	(9,417)	(15,754)	(7,225)
Other operating expenses	(8,498)	(8,737)	(7,717)
Total non-interest expense	(128,972)	(186,146)	(90,685)
Income (loss) before provision for income taxes	56,898	54,119	(4,261)
Total assets	482,185	626,035	330,363
Corporate
Segment reporting
Interest expense	(2,699)	(1,271)
Net interest income before provision for loan losses	(2,699)	(1,271)
Net interest income after provision for (recovery of) loan losses	(2,699)	(1,271)
Non-interest income
Net unrealized gain (loss) on financial instruments			(253)
Other income (loss)	85	189	164
Gain on bargain purchase			30,728
Total non-interest income (loss)	85	189	30,639
Non-interest expense
Employee compensation and benefits	(3,753)	(3,407)	(3,340)
Allocated employee compensation and benefits from related party	(10,828)	(6,300)	(4,926)
Professional fees	(6,290)	(7,570)	(2,947)
Management fees - related party	(10,928)	(10,682)	(9,578)
Incentive fees - related party	(5,419)	(5,973)	(106)
Loan servicing expense		(37)	(133)
Transaction related expenses	(14,282)	(63)	(7,750)
Other operating expenses	(4,529)	(3,242)	(3,082)
Total non-interest expense	(56,029)	(37,274)	(31,862)
Income (loss) before provision for income taxes	(58,643)	(38,356)	(1,223)
Total assets	$ 386,999	$ 200,561	$ 130,242