Secured Borrowings (Details) $ in Thousands, € in Millions	1 Months Ended	6 Months Ended
	Jun. 30, 2020 USD ($)	Jun. 30, 2020 USD ($)	Jun. 30, 2020 EUR (€)	Dec. 31, 2019 USD ($)
Secured borrowings and promissory note
Current facility size	$ 2,683,188	$ 2,683,188
Pledged Assets Carrying Value	1,668,889	1,668,889		$ 1,668,460
Carrying Value, Secured borrowings	1,253,895	1,253,895		1,189,392
Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	1,199,180	1,199,180
Pledged Assets Carrying Value	628,903	628,903		561,257
Carrying Value, Secured borrowings	$ 539,733	$ 539,733		$ 380,203
Weighted average interest rate of borrowings (as a percent)	3.20%	3.20%	3.20%	4.00%
Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 1,484,008	$ 1,484,008
Pledged Assets Carrying Value	1,039,986	1,039,986		$ 1,107,203
Carrying Value, Secured borrowings	$ 714,162	$ 714,162		$ 809,189
Weighted average interest rate of borrowings (as a percent)	3.70%	3.70%	3.70%	4.20%
PPP Loans
Secured borrowings and promissory note
Carrying Value, Secured borrowings	$ 100,012	$ 100,012
PPP Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Pledged Assets Carrying Value	105,248	105,248
Purchased future receivables | Borrowings under credit facilities
Secured borrowings and promissory note
Pledged Assets Carrying Value	27,190	27,190		$ 43,265
Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Pledged Assets Carrying Value	71,546	71,546		113,436
Waterfall Victoria Master Fund II | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	35,000	35,000
Pledged Assets Carrying Value	61,200	61,200
Carrying Value, Secured borrowings	$ 35,000	35,000
Waterfall Victoria Master Fund II | Borrowings under repurchase agreements | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	12.00%
Waterfall Victoria Master Fund II | Originated SBC, originated transitional, and acquired loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 35,000	35,000
Pledged Assets Carrying Value	61,226	61,226
Carrying Value, Secured borrowings	35,000	35,000
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	250,000	250,000
Pledged Assets Carrying Value	47,576	47,576
Carrying Value, Secured borrowings	34,525	$ 34,525		88,972
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.00%
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.75%
JPMorgan | Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	400,000	$ 400,000
Pledged Assets Carrying Value	269,399	269,399
Carrying Value, Secured borrowings	182,293	$ 182,293		250,466
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.25%
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		4.00%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	81,093	$ 81,093
Pledged Assets Carrying Value	118,452	118,452
Carrying Value, Secured borrowings	81,093	$ 81,093		93,715
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.82%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		4.82%
Keybank | Freddie Mac Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	100,000	$ 100,000
Pledged Assets Carrying Value	15,355	15,355
Carrying Value, Secured borrowings	15,071	$ 15,071		21,513
Keybank | Freddie Mac Loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.30%
East West Bank | SBA loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	50,000	$ 50,000
Pledged Assets Carrying Value	43,572	43,572
Carrying Value, Secured borrowings	33,887	$ 33,887		13,294
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		(0.821%)
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		0.29%
East West Bank | Residential MSRs | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	50,000	$ 50,000
Pledged Assets Carrying Value	49,699	49,699
Carrying Value, Secured borrowings	37,700	$ 37,700		39,900
East West Bank | Residential MSRs | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.50%
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	224,680	$ 224,680	€ 200.0
Pledged Assets Carrying Value	51,025	51,025
Carrying Value, Secured borrowings	37,709	$ 37,709		37,646
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities | Euribor Rate
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.50%
Credit Suisse | Purchased future receivables | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		4.50%
Credit Suisse | Purchased future receivables | PPP Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	150,000	$ 150,000
Pledged Assets Carrying Value	132,438	132,438
Carrying Value, Secured borrowings	$ 114,345	$ 114,345		34,900
FCB | Acquired loans | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	2.75%	2.75%	2.75%
Carrying Value, Secured borrowings				1,354
Comerica Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 100,000	$ 100,000
Pledged Assets Carrying Value	92,811	92,811
Carrying Value, Secured borrowings	85,930	$ 85,930		56,822
Comerica Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.75%
TBK Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	150,000	$ 150,000
Pledged Assets Carrying Value	129,298	129,298
Carrying Value, Secured borrowings	123,459	123,459		52,151
Origin Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	70,000	70,000
Pledged Assets Carrying Value	26,079	26,079
Carrying Value, Secured borrowings	24,277	24,277		15,343
Associated Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	40,000	40,000
Pledged Assets Carrying Value	22,097	22,097
Carrying Value, Secured borrowings	$ 20,477	$ 20,477		5,823
Associated Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.50%
Rabobank | Real estate | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.22%	4.22%	4.22%
Current facility size	$ 14,500	$ 14,500
Pledged Assets Carrying Value	18,953	18,953
Carrying Value, Secured borrowings	12,353	12,353		12,485
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	500,000	500,000
Pledged Assets Carrying Value	158,192	158,192
Carrying Value, Secured borrowings	132,257	$ 132,257		124,718
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.875%
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.125%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	61,244	$ 61,244
Pledged Assets Carrying Value	114,749	114,749
Carrying Value, Secured borrowings	61,244	$ 61,244		56,189
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		4.37%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		5.60%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	350,000	$ 350,000
Pledged Assets Carrying Value	237,295	237,295
Carrying Value, Secured borrowings	165,604	$ 165,604		141,356
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.00%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		2.40%
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	16,354	$ 16,354
Pledged Assets Carrying Value	20,189	20,189
Carrying Value, Secured borrowings	16,354	$ 16,354		44,730
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		3.54%
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Carrying Value, Secured borrowings				38,954
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.31%
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		1.61%
RBC | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	40,317	$ 40,317
Pledged Assets Carrying Value	60,484	60,484
Carrying Value, Secured borrowings	$ 40,317	$ 40,317		$ 59,061
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		3.03%
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)		4.43%