Secured Borrowings (Details) $ in Thousands, € in Millions	3 Months Ended
	Mar. 31, 2020 EUR (€)	Mar. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)
Secured borrowings and promissory note
Current facility size		$ 2,689,635
Pledged Assets Carrying Value		2,162,870	$ 1,668,460
Carrying Value, Secured borrowings		1,698,937	1,189,392
Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		1,080,218
Pledged Assets Carrying Value		613,793	561,257
Carrying Value, Secured borrowings		$ 539,580	$ 380,203
Weighted average interest rate of borrowings (as a percent)	2.90%	2.90%	4.00%
Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		$ 1,609,417
Pledged Assets Carrying Value		1,549,077	$ 1,107,203
Carrying Value, Secured borrowings		$ 1,159,357	$ 809,189
Weighted average interest rate of borrowings (as a percent)	3.40%	3.40%	4.20%
Purchased future receivables | Borrowings under credit facilities
Secured borrowings and promissory note
Pledged Assets Carrying Value		$ 49,150	$ 43,265
Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Pledged Assets Carrying Value		74,439	113,436
Lender 1
Secured borrowings and promissory note
Carrying Value, Secured borrowings		312,020	312,019
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		250,000
Pledged Assets Carrying Value		104,668
Carrying Value, Secured borrowings		82,204	88,972
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.00%
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
JPMorgan | Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		400,000
Pledged Assets Carrying Value		423,990
Carrying Value, Secured borrowings		297,342	250,466
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.25%
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.00%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		90,507
Pledged Assets Carrying Value		122,070
Carrying Value, Secured borrowings		90,507	93,715
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.87%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.52%
Keybank | Freddie Mac Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		100,000
Pledged Assets Carrying Value		79,524
Carrying Value, Secured borrowings		78,726	21,513
Keybank | Freddie Mac Loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.30%
East West Bank | SBA loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		50,000
Pledged Assets Carrying Value		39,285
Carrying Value, Secured borrowings		30,911	13,294
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	(0.821%)
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	0.29%
East West Bank | Residential MSRs | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		50,000
Pledged Assets Carrying Value		53,044
Carrying Value, Secured borrowings		49,200	39,900
East West Bank | Residential MSRs | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	€ 200.0	224,500
Pledged Assets Carrying Value		51,274
Carrying Value, Secured borrowings		37,030	37,646
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities | Euribor Rate
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Credit Suisse | Purchased future receivables | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		50,000
Pledged Assets Carrying Value		49,150
Carrying Value, Secured borrowings		$ 43,250	34,900
Credit Suisse | Purchased future receivables | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.50%
FCB | Acquired loans | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	2.75%	2.75%
Current facility size		$ 1,218
Pledged Assets Carrying Value		1,737
Carrying Value, Secured borrowings		1,218	1,354
Comerica Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		100,000
Pledged Assets Carrying Value		82,991
Carrying Value, Secured borrowings		77,926	56,822
Comerica Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.75%
TBK Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		150,000
Pledged Assets Carrying Value		78,585
Carrying Value, Secured borrowings		76,358	52,151
Origin Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		50,000
Pledged Assets Carrying Value		17,885
Carrying Value, Secured borrowings		16,143	15,343
Associated Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size		40,000
Pledged Assets Carrying Value		36,697
Carrying Value, Secured borrowings		$ 34,228	5,823
Associated Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.50%
Rabobank | Real estate | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.22%	4.22%
Current facility size		$ 14,500
Pledged Assets Carrying Value		18,953
Carrying Value, Secured borrowings		12,386	12,485
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		500,000
Pledged Assets Carrying Value		305,046
Carrying Value, Secured borrowings		246,417	124,718
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.875%
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.125%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		55,275
Pledged Assets Carrying Value		76,566
Carrying Value, Secured borrowings		55,275	56,189
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.89%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.01%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		425,000
Pledged Assets Carrying Value		472,064
Carrying Value, Secured borrowings		331,181	141,356
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.00%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.40%
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		47,612
Pledged Assets Carrying Value		52,173
Carrying Value, Secured borrowings		47,612	44,730
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.14%
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.81%
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		34,886
Pledged Assets Carrying Value		33,968
Carrying Value, Secured borrowings		34,886	38,954
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.31%
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.61%
RBC | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size		56,137
Pledged Assets Carrying Value		63,200
Carrying Value, Secured borrowings		$ 56,137	$ 59,061
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.03%
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.71%