Other Assets and Liabilities - REO Information (Details) $ in Thousands	3 Months Ended	12 Months Ended
	Sep. 30, 2019 partnership	Dec. 31, 2020 USD ($) asset	Dec. 31, 2019 USD ($) asset	Dec. 31, 2018 partnership
Other Real Estate [Roll Forward]
Balance at beginning of period		$ 9,462
Transfers to REO		14,119
Liquidations		(14,622)
Changes in fair value, net		(546)
Balance at End of Period		$ 8,413	$ 9,462
Number of REO properties recorded on balance sheet | asset		17	13
Number of partnerships consolidated | partnership	2			2
Payable to minority partner		$ 16,941	$ 13,189
Co-investors gains (losses)		(200)	1,000
Legacy Sequoia
Other Real Estate [Roll Forward]
Balance at beginning of period		460
Transfers to REO		532
Liquidations		(243)
Changes in fair value, net		(111)
Balance at End of Period		$ 638	$ 460
Number of REO properties recorded on balance sheet | asset		3	4
Freddie Mac SLST
Other Real Estate [Roll Forward]
Balance at beginning of period		$ 445
Transfers to REO		1,319
Liquidations		(1,178)
Changes in fair value, net		60
Balance at End of Period		$ 646	$ 445
Number of REO properties recorded on balance sheet | asset		9	3
CAFL
Other Real Estate [Roll Forward]
Balance at beginning of period		$ 1,670
Transfers to REO		6,157
Liquidations		(4,371)
Changes in fair value, net		(927)
Balance at End of Period		$ 2,529	$ 1,670
Number of REO properties recorded on balance sheet | asset		2	2
Bridge Loan
Other Real Estate [Roll Forward]
Balance at beginning of period		$ 6,887
Transfers to REO		6,111
Liquidations		(8,830)
Changes in fair value, net		432
Balance at End of Period		$ 4,600	$ 6,887
Number of REO properties recorded on balance sheet | asset		3	4
Debt Securities
Other Real Estate [Roll Forward]
Liquidations		$ (1,000)