Loans and Allowance for Loan Losses - Credit Quality (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2019	Dec. 31, 2018
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 3,311,156	$ 2,416,303
General allowance for loan losses	(2,264)	(1,937)
Total Loans, net of allowance for loan losses, including loans in consolidated VIEs	3,308,892	2,414,366
Carrying amount of loan foreclosure in process	$ 700	$ 1,400
Percentage of outstanding	100.00%	100.00%
0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 235,620	$ 124,928
Percentage of outstanding	7.10%	5.20%
20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 476,080	$ 293,021
Percentage of outstanding	14.40%	12.10%
40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 914,495	$ 697,709
Percentage of outstanding	27.60%	28.90%
60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 1,462,372	$ 1,001,900
Percentage of outstanding	44.20%	41.50%
80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 137,252	$ 200,209
Percentage of outstanding	4.10%	8.30%
Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 85,337	$ 98,536
Percentage of outstanding	2.60%	4.00%
Acquired SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 106,668	$ 318,488
General allowance for loan losses	(347)	(532)
Acquired SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	3,469	6,337
Acquired SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	17,402	38,150
Acquired SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	23,759	100,578
Acquired SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	24,030	93,411
Acquired SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	20,156	33,750
Acquired SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	17,852	46,262
Acquired loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	967,995	545,695
General allowance for loan losses	(780)	(608)
Acquired loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	225,178	118,198
Acquired loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	335,958	165,567
Acquired loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	253,597	136,206
Acquired loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	103,903	70,017
Acquired loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	23,294	40,003
Acquired loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	26,065	15,704
Acquired Transitional loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	83,355
General allowance for loan losses	(62)
Acquired Transitional loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	5,889
Acquired Transitional loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	10,038
Acquired Transitional loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	31,180
Acquired Transitional loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	27,112
Acquired Transitional loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	9,136
Originated Transitional loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	899,179	664,733
General allowance for loan losses	(290)	(353)
Originated Transitional loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	852
Originated Transitional loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	47,601	29,245
Originated Transitional loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	135,362	178,861
Originated Transitional loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	660,448	348,967
Originated Transitional loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	54,863	101,513
Originated Transitional loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	53	6,147
Originated SBC loans, at fair value
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	20,434	22,664
Originated SBC loans, at fair value | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	8,558	8,600
Originated SBC loans, at fair value | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	10,349	6,328
Originated SBC loans, at fair value | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,527	7,736
Originated SBC loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,119,872	777,408
General allowance for loan losses	(23)	(11)
Originated SBC loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	51,508	48,259
Originated SBC loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	456,028	271,311
Originated SBC loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	595,649	457,838
Originated SBC loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	10,654
Originated SBC loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	6,033
Originated SBA 7(a) loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	110,784	85,416
General allowance for loan losses	(762)	(433)
Originated SBA 7(a) loans | 0.0 - 20.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	232	393
Originated SBA 7(a) loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	4,964	3,200
Originated SBA 7(a) loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	14,531	10,642
Originated SBA 7(a) loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	40,048	24,387
Originated SBA 7(a) loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	16,017	16,473
Originated SBA 7(a) loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	34,992	30,321
Originated Residential Agency loans
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	2,869	1,899
Originated Residential Agency loans | 20.1 - 40.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	51
Originated Residential Agency loans | 40.1 - 60.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	38	111
Originated Residential Agency loans | 60.1 - 80.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	833	952
Originated Residential Agency loans | 80.1 - 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	1,605	734
Originated Residential Agency loans | Greater than 100.0%
Credit quality of loans
Total Loans, before allowance for loan losses, including loans in consolidated VIEs	$ 342	$ 102