Loans and Allowance for Loan Losses - Credit Quality (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 4,127,901	$ 4,056,607
General allowance for loan losses	(29,539)	(2,424)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 4,098,362	$ 4,054,183
Percentage of loans outstanding	100.00%	100.00%
Carrying amount of loan foreclosure in process	$ 2,200	$ 800
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 285,928	$ 228,992
Percentage of loans outstanding	6.90%	5.60%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 532,200	$ 521,832
Percentage of loans outstanding	12.90%	12.90%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,075,306	$ 1,146,672
Percentage of loans outstanding	26.10%	28.30%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,728,645	$ 1,793,845
Percentage of loans outstanding	41.90%	44.20%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 289,919	$ 198,096
Percentage of loans outstanding	7.00%	4.90%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 215,903	$ 167,170
Percentage of loans outstanding	5.20%	4.10%
Originated Transitional loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,319,074	$ 1,086,328
General allowance for loan losses	(14,995)	(64)
Originated Transitional loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,485	1,736
Originated Transitional loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	8,269	28,108
Originated Transitional loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	252,798	277,388
Originated Transitional loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	891,895	750,298
Originated Transitional loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	157,900	28,059
Originated Transitional loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,727	739
Originated SBC loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,056,555	1,168,899
General allowance for loan losses	(2,640)	(124)
Originated SBC loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,372
Originated SBC loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	76,899	60,601
Originated SBC loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	453,381	431,312
Originated SBC loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	515,023	660,733
Originated SBC loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		8,045
Originated SBC loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,880	8,208
Acquired loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,046,109	1,094,283
General allowance for loan losses	(5,457)	(803)
Acquired loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	266,345	218,679
Acquired loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	385,579	371,471
Acquired loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	228,262	293,216
Acquired loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	113,023	161,431
Acquired loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	40,838	35,731
Acquired loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	12,062	13,755
Originated SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	374,791	376,629
General allowance for loan losses	(5,680)	(1,019)
Originated SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,203	865
Originated SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	15,013	13,843
Originated SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	51,133	41,166
Originated SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	147,020	130,177
Originated SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	61,297	78,544
Originated SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	99,125	112,034
Acquired SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	239,438	306,860
General allowance for loan losses	(767)	(414)
Acquired SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	7,523	7,712
Acquired SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	39,086	39,566
Acquired SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	89,644	103,590
Acquired SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	54,007	83,954
Acquired SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	28,332	39,726
Acquired SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	20,846	32,312
Originated PPP loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	74,931
Originated PPP loans, at fair value | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	74,931
Originated SBC loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	13,795	20,212
Originated SBC loans, at fair value | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	7,354	8,192
Originated SBC loans, at fair value | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	6,441	6,422
Originated SBC loans, at fair value | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		5,598
Originated Residential Agency loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,208	3,396
Originated Residential Agency loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		51
Originated Residential Agency loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	88
Originated Residential Agency loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,236	830
Originated Residential Agency loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,552	2,393
Originated Residential Agency loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 332	$ 122