Segment Reporting (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2021 USD ($) segment	Mar. 31, 2020 USD ($) segment
Segment reporting
Number of reportable segments | segment	4	4
Interest income	$ 73,371	$ 69,551
Interest expense	(50,761)	(46,930)
Net interest income before provision for loan losses	22,610	22,621
Provision for (recovery of) loan losses	8	(39,804)
Net interest income after provision for (recovery of) loan losses	22,618	(17,183)
Non-interest income
Residential mortgage banking activities	41,409	36,669
Net realized gains on financial instruments and real estate owned	8,846	7,172
Net unrealized gain (loss) on financial instruments	20,996	(33,434)
Other income	571	4,073
Servicing income	15,635	8,097
Income on purchased future receivables, net of allowance for doubtful accounts	2,317	3,483
Income (Loss) from unconsolidated joint ventures	(809)	(3,537)
Total non-interest income (loss)	88,965	22,523
Non-interest expense
Employee compensation and benefits	(22,777)	(18,936)
Allocated employee compensation and benefits from related party	(2,123)	(1,250)
Variable expenses on residential mortgage banking activities	(15,485)	(20,129)
Professional fees	(2,982)	(2,556)
Management fees - related party	(2,693)	(2,561)
Loan servicing expense	(6,104)	(5,570)
Merger related expenses	(6,307)	(47)
Other operating expenses	(15,484)	(13,744)
Total non-interest expense	(73,955)	(64,793)
Income (loss) before provision for income taxes	37,628	(59,453)
Total assets	8,016,955	5,270,050
Corporate- Other
Segment reporting
Interest income	1,668
Interest expense	(2,257)
Net interest income before provision for loan losses	(589)
Net interest income after provision for (recovery of) loan losses	(589)
Non-interest income
Net realized gains on financial instruments and real estate owned	(126)
Net unrealized gain (loss) on financial instruments	1,197
Other income	45	99
Total non-interest income (loss)	1,116	99
Non-interest expense
Employee compensation and benefits	(891)	(742)
Allocated employee compensation and benefits from related party	(1,911)	(1,125)
Professional fees	(1,242)	(1,407)
Management fees - related party	(2,693)	(2,561)
Loan servicing expense	(39)	(32)
Merger related expenses	(6,307)	(47)
Other operating expenses	(715)	(698)
Total non-interest expense	(13,798)	(6,612)
Income (loss) before provision for income taxes	(13,271)	(6,513)
Total assets	1,111,088	234,380
Loan Acquisitions | Operating Segments
Segment reporting
Interest income	14,534	16,494
Interest expense	(11,971)	(11,205)
Net interest income before provision for loan losses	2,563	5,289
Provision for (recovery of) loan losses	1,262	(5,722)
Net interest income after provision for (recovery of) loan losses	3,825	(433)
Non-interest income
Net realized gains on financial instruments and real estate owned	(1,493)	(739)
Net unrealized gain (loss) on financial instruments	897	(9,423)
Other income	1,183	2,336
Servicing income	21	355
Income on purchased future receivables, net of allowance for doubtful accounts	2,317	3,483
Income (Loss) from unconsolidated joint ventures	(809)	(3,537)
Total non-interest income (loss)	2,116	(7,525)
Non-interest expense
Employee compensation and benefits	(1,485)	(2,833)
Allocated employee compensation and benefits from related party	(212)	(125)
Professional fees	(786)	(235)
Loan servicing expense	(1,751)	(1,365)
Other operating expenses	(2,364)	(6,245)
Total non-interest expense	(6,598)	(10,803)
Income (loss) before provision for income taxes	(657)	(18,761)
Total assets	1,121,590	1,209,617
SBC Originations | Operating Segments
Segment reporting
Interest income	39,693	39,269
Interest expense	(24,998)	(25,627)
Net interest income before provision for loan losses	14,695	13,642
Provision for (recovery of) loan losses	(1,609)	(29,828)
Net interest income after provision for (recovery of) loan losses	13,086	(16,186)
Non-interest income
Net realized gains on financial instruments and real estate owned	5,565	3,649
Net unrealized gain (loss) on financial instruments	3,033	(6,491)
Other income	1,288	1,283
Servicing income	726	532
Total non-interest income (loss)	10,612	(1,027)
Non-interest expense
Employee compensation and benefits	(2,252)	(2,710)
Professional fees	(323)	(338)
Loan servicing expense	(2,052)	(1,580)
Other operating expenses	(3,916)	(3,457)
Total non-interest expense	(8,543)	(8,085)
Income (loss) before provision for income taxes	15,155	(25,298)
Total assets	3,128,924	2,704,301
SBA Originations, Acquisitions, and Servicing | Operating Segments
Segment reporting
Interest income	15,432	12,471
Interest expense	(9,207)	(8,513)
Net interest income before provision for loan losses	6,225	3,958
Provision for (recovery of) loan losses	355	(4,254)
Net interest income after provision for (recovery of) loan losses	6,580	(296)
Non-interest income
Net realized gains on financial instruments and real estate owned	4,900	4,262
Net unrealized gain (loss) on financial instruments	514	(1,082)
Other income	(1,960)	295
Servicing income	7,782	1,074
Total non-interest income (loss)	11,236	4,549
Non-interest expense
Employee compensation and benefits	(4,561)	(3,910)
Professional fees	(380)	(289)
Loan servicing expense	102	(335)
Other operating expenses	(6,285)	(1,559)
Total non-interest expense	(11,124)	(6,093)
Income (loss) before provision for income taxes	6,692	(1,840)
Total assets	1,983,098	703,331
Residential Mortgage Banking | Operating Segments
Segment reporting
Interest income	2,044	1,317
Interest expense	(2,328)	(1,585)
Net interest income before provision for loan losses	(284)	(268)
Net interest income after provision for (recovery of) loan losses	(284)	(268)
Non-interest income
Residential mortgage banking activities	41,409	36,669
Net unrealized gain (loss) on financial instruments	15,355	(16,438)
Other income	15	60
Servicing income	7,106	6,136
Total non-interest income (loss)	63,885	26,427
Non-interest expense
Employee compensation and benefits	(13,588)	(8,741)
Variable expenses on residential mortgage banking activities	(15,485)	(20,129)
Professional fees	(251)	(287)
Loan servicing expense	(2,364)	(2,258)
Other operating expenses	(2,204)	(1,785)
Total non-interest expense	(33,892)	(33,200)
Income (loss) before provision for income taxes	29,709	(7,041)
Total assets	$ 672,255	$ 418,421