Loans and Allowance for Loan Losses - Credit Quality (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2020	Dec. 31, 2019
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 4,197,950	$ 4,056,607
General allowance for loan losses	(42,969)	(2,424)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	$ 4,154,981	$ 4,054,183
Percentage of outstanding	100.00%	100.00%
Carrying amount of loan foreclosure in process	$ 1,100	$ 800
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 238,076	$ 228,992
Percentage of outstanding	5.70%	5.60%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 541,815	$ 521,832
Percentage of outstanding	12.90%	12.90%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,068,627	$ 1,146,672
Percentage of outstanding	25.50%	28.30%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,834,849	$ 1,793,845
Percentage of outstanding	43.70%	44.20%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 253,937	$ 198,096
Percentage of outstanding	6.00%	4.90%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 260,646	$ 167,170
Percentage of outstanding	6.20%	4.10%
Originated Transitional loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,268,775	$ 1,086,328
Originated Transitional loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,407	1,736
Originated Transitional loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	35,443	28,108
Originated Transitional loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	223,154	277,388
Originated Transitional loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	897,430	750,298
Originated Transitional loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	104,974	28,059
Originated Transitional loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,367	739
Originated SBC loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,174,414	1,168,899
Originated SBC loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	66,041	60,601
Originated SBC loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	463,824	431,312
Originated SBC loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	613,058	660,733
Originated SBC loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	22,754	8,045
Originated SBC loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	8,737	8,208
Acquired loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	997,830	1,094,283
Acquired loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	222,563	218,679
Acquired loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	378,245	371,471
Acquired loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	236,043	293,216
Acquired loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	113,572	161,431
Acquired loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	28,988	35,731
Acquired loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	18,419	13,755
Acquired SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	266,138	306,860
Acquired SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	8,741	7,712
Acquired SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	38,950	39,566
Acquired SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	96,622	103,590
Acquired SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	65,475	83,954
Acquired SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	30,392	39,726
Acquired SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	25,958	32,312
Originated PPP loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	105,248
Originated PPP loans, at fair value | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	105,248
Originated SBC loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	19,050	20,212
Originated SBC loans, at fair value | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	7,591	8,192
Originated SBC loans, at fair value | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	11,459	6,422
Originated SBC loans, at fair value | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs		5,598
Originated SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	362,200	376,629
Originated SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,365	865
Originated SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	15,494	13,843
Originated SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	48,894	41,166
Originated SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	133,045	130,177
Originated SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	63,760	78,544
Originated SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	99,642	112,034
Originated Residential Agency loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,295	3,396
Originated Residential Agency loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	51	51
Originated Residential Agency loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	90
Originated Residential Agency loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	810	830
Originated Residential Agency loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,069	2,393
Originated Residential Agency loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 275	$ 122