Loans and Allowance for Credit Losses - Credit Quality (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2021	Dec. 31, 2020
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 4,517,066	$ 4,052,970
General allowance for loan losses	(27,403)	(29,539)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 4,489,663	$ 4,023,431
Percentage of loans outstanding	100.00%	100.00%
Carrying amount of loan foreclosure in process	$ 1,100	$ 2,200
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 285,667	$ 285,928
Percentage of loans outstanding	6.30%	7.10%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 515,154	$ 532,200
Percentage of loans outstanding	11.40%	13.00%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,109,080	$ 1,075,306
Percentage of loans outstanding	24.60%	26.50%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 2,080,943	$ 1,728,645
Percentage of loans outstanding	46.10%	42.70%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 363,683	$ 289,919
Percentage of loans outstanding	8.10%	7.20%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 162,539	$ 140,972
Percentage of loans outstanding	3.50%	3.50%
Originated Transitional loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,873,164	$ 1,319,074
General allowance for loan losses	(14,848)	(14,995)
Originated Transitional loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	11,669	5,485
Originated Transitional loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	11,319	8,269
Originated Transitional loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	237,489	252,798
Originated Transitional loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,359,157	891,895
Originated Transitional loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	236,062	157,900
Originated Transitional loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	17,468	2,727
Originated SBC loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,039,537	1,056,555
General allowance for loan losses	(2,729)	(2,640)
Originated SBC loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,248	5,372
Originated SBC loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	73,432	76,899
Originated SBC loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	520,800	453,381
Originated SBC loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	428,041	515,023
Originated SBC loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	6,136
Originated SBC loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,880	5,880
Acquired loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	990,301	1,046,109
General allowance for loan losses	(4,335)	(5,457)
Acquired loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	261,037	266,345
Acquired loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	373,055	385,579
Acquired loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	224,470	228,262
Acquired loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	94,011	113,023
Acquired loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	25,307	40,838
Acquired loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	12,421	12,062
Originated SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	372,288	374,791
General allowance for loan losses	(4,864)	(5,680)
Originated SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,138	1,203
Originated SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	15,519	15,013
Originated SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	46,789	51,133
Originated SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	138,676	147,020
Originated SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	66,520	61,297
Originated SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	103,646	99,125
Acquired SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	225,020	239,438
General allowance for loan losses	(627)	(767)
Acquired SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	6,575	7,523
Acquired SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	33,186	39,086
Acquired SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	79,334	89,644
Acquired SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	55,506	54,007
Acquired SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	27,673	28,332
Acquired SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	22,746	20,846
Originated SBC loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	13,618	13,795
Originated SBC loans, at fair value | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	8,643	7,354
Originated SBC loans, at fair value | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,975	6,441
Originated Residential Agency loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,138	3,208
Originated Residential Agency loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	198	88
Originated Residential Agency loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	577	1,236
Originated Residential Agency loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,985	1,552
Originated Residential Agency loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 378	$ 332