Secured Borrowings (Details) $ in Thousands, € in Millions	12 Months Ended
	Dec. 31, 2020 USD ($)	Dec. 31, 2020 EUR (€)	Dec. 31, 2019 USD ($)
Secured borrowings and promissory note
Carrying Value, Secured borrowings	$ 1,370,519		$ 1,189,392
Secured borrowings
Secured borrowings and promissory note
Current facility size	2,760,403
Pledged Assets Carrying Value	1,767,319		1,668,460
Carrying Value, Secured borrowings	1,370,519		1,189,392
Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	1,331,752
Pledged Assets Carrying Value	631,014		561,257
Carrying Value, Secured borrowings	$ 542,950		$ 380,203
Weighted average interest rate of borrowings (as a percent)	2.80%	2.80%	4.00%
Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 1,428,651
Pledged Assets Carrying Value	1,136,305		$ 1,107,203
Carrying Value, Secured borrowings	$ 827,569		$ 809,189
Weighted average interest rate of borrowings (as a percent)	3.30%	3.30%	4.20%
PPP Loans
Secured borrowings and promissory note
Carrying Value, Secured borrowings	$ 76,276
Purchased future receivables | Borrowings under credit facilities
Secured borrowings and promissory note
Pledged Assets Carrying Value			$ 43,265
Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Pledged Assets Carrying Value	72,179		113,436
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	200,000
Pledged Assets Carrying Value	52,068
Carrying Value, Secured borrowings	$ 36,604		88,972
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.25%
JPMorgan | Acquired SBA Loans | Borrowings under credit facilities | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.875%
JPMorgan | Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 400,000
Pledged Assets Carrying Value	375,035
Carrying Value, Secured borrowings	$ 247,616		250,466
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.25%
JPMorgan | Transitional loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.00%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 65,407
Pledged Assets Carrying Value	107,347
Carrying Value, Secured borrowings	$ 65,407		93,715
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.54%
JPMorgan | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.75%
Keybank | Freddie Mac Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 100,000
Pledged Assets Carrying Value	51,248
Carrying Value, Secured borrowings	$ 50,408		21,513
Keybank | Freddie Mac Loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.30%
East West Bank | SBA loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 50,000
Pledged Assets Carrying Value	50,516
Carrying Value, Secured borrowings	$ 40,542		13,294
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	(0.821%)
East West Bank | SBA loans | Borrowings under credit facilities | Prime rate | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	0.29%
East West Bank | Residential MSRs | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 50,000
Pledged Assets Carrying Value	50,941
Carrying Value, Secured borrowings	$ 34,400		39,900
East West Bank | Residential MSRs | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 244,280	€ 200.0
Pledged Assets Carrying Value	59,353
Carrying Value, Secured borrowings	$ 36,840		37,646
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities | Euribor Rate | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Credit Suisse | Acquired loans (non USD) | Borrowings under credit facilities | Euribor Rate | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.00%
Credit Suisse | Purchased future receivables | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 150,000
Carrying Value, Secured borrowings			$ 34,900
Credit Suisse | Purchased future receivables | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.50%
FCB | Acquired loans | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)			2.75%
Carrying Value, Secured borrowings			$ 1,354
Comerica Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 125,000
Pledged Assets Carrying Value	84,755
Carrying Value, Secured borrowings	$ 78,312		56,822
Comerica Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.75%
TBK Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	$ 150,000
Pledged Assets Carrying Value	129,043
Carrying Value, Secured borrowings	123,951		52,151
Origin Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	60,000
Pledged Assets Carrying Value	29,381
Carrying Value, Secured borrowings	27,450		15,343
Associated Bank | Residential loans | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	60,000
Pledged Assets Carrying Value	16,962
Carrying Value, Secured borrowings	$ 15,556		5,823
Associated Bank | Residential loans | Borrowings under credit facilities | One Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.50%
Rabobank | Real estate | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.22%	4.22%
Current facility size	$ 14,500
Carrying Value, Secured borrowings			12,485
Federal Reserve Bank of Minneapolis | PPP Loans | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	0.35%	0.35%
Current facility size	$ 105,222
Pledged Assets Carrying Value	73,799
Carrying Value, Secured borrowings	76,276
Bank of the Sierra | Real estate | Borrowings under credit facilities
Secured borrowings and promissory note
Current facility size	22,750
Pledged Assets Carrying Value	32,948
Carrying Value, Secured borrowings	$ 22,611
Bank of the Sierra | Real estate | Borrowings under credit facilities | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.25%	3.25%
Bank of the Sierra | Real estate | Borrowings under credit facilities | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.45%	3.45%
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 500,000
Pledged Assets Carrying Value	196,304
Carrying Value, Secured borrowings	$ 210,735		124,718
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Citibank | Fixed rate, Transitional, Acquired loans | Borrowings under repurchase agreements | One Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.25%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 58,076
Pledged Assets Carrying Value	111,796
Carrying Value, Secured borrowings	$ 58,076		56,189
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.25%
Citibank | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.75%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	$ 350,000
Pledged Assets Carrying Value	266,014
Carrying Value, Secured borrowings	$ 190,567		141,356
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Minimum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.00%
Deutsche Bank | Fixed rate, Transitional loans | Borrowings under repurchase agreements | Three Month LIBOR | Maximum
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.40%
Deutsche Bank | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	3.54%
Current facility size	$ 16,354
Pledged Assets Carrying Value	20,189
Carrying Value, Secured borrowings	16,354		44,730
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Carrying Value, Secured borrowings			$ 38,954
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)			1.31%
Bank of America | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)			1.61%
RBC | Mortgage backed securities | Borrowings under repurchase agreements
Secured borrowings and promissory note
Current facility size	38,814
Pledged Assets Carrying Value	59,620
Carrying Value, Secured borrowings	$ 38,814		$ 59,061
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.05%	3.05%
RBC | Mortgage backed securities | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.43%	4.43%