Fair Value of Financial Instruments - Financial Instruments Measured at Level III (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	$ (1,540)	$ (5,141)	$ (7,398)
Other securities
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(1,574)	(4,781)	(6,472)
Level III | Fair Value, Measurements, Recurring | Securitized Debt
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	15,418	1,057
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	0	0
Sales and settlements		0
Transfers to REO	0	0
VIE consolidation		17,960
VIE deconsolidation		0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	0	0
Unrealized gains (losses), net on liabilities	4,056	(887)
Premium and discount amortization, net	(4,555)	(2,712)
Ending balance	14,919	15,418	1,057
Unrealized gain (loss)	(3,603)	887
Level III | Fair Value, Measurements, Recurring | Securitized Debt | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Residential Whole Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,008,782	1,375,860
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	422,041	92,822
Sales and settlements		(144,259)
Transfers to REO	0	0
VIE consolidation		0
VIE deconsolidation		0
Loan modifications / capitalized interest	486	779
Principal repayments	(401,075)	(278,316)
Realized gains/(losses), net on assets	0	(10,511)
Unrealized gains/(losses), net on assets	2,354	(23,094)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	(9,086)	(4,499)
Ending balance	1,023,502	1,008,782	1,375,860
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Residential Whole Loans | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	5,795	(14,807)
Level III | Fair Value, Measurements, Recurring | Residential Bridge Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	12,813	33,269
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	0	0
Sales and settlements		0
Transfers to REO	(752)	(419)
VIE consolidation		0
VIE deconsolidation		0
Loan modifications / capitalized interest	0	0
Principal repayments	(7,312)	(19,105)
Realized gains/(losses), net on assets	(206)	(373)
Unrealized gains/(losses), net on assets	907	(499)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	(22)	(60)
Ending balance	5,428	12,813	33,269
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Residential Bridge Loans | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	149	(881)
Level III | Fair Value, Measurements, Recurring | Commercial Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	310,523	370,213
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	0	0
Sales and settlements		0
Transfers to REO	(30,000)	0
VIE consolidation		0
VIE deconsolidation		0
Loan modifications / capitalized interest	0	49
Principal repayments	(103,285)	(44,819)
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	(46,813)	(15,282)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	147	362
Ending balance	130,572	310,523	370,213
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Commercial Loans | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	(50,034)	(15,282)
Level III | Fair Value, Measurements, Recurring | Securitized Commercial Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,605,335	909,040
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	0	0
Sales and settlements		0
Transfers to REO	0	0
VIE consolidation		1,245,287
VIE deconsolidation		(150,804)
Loan modifications / capitalized interest	0	0
Principal repayments	(354,202)	(349,609)
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	79,972	(58,421)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	24,703	9,842
Ending balance	1,355,808	1,605,335	909,040
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Securitized Commercial Loans | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	64,973	746
Level III | Fair Value, Measurements, Recurring | Subtotal Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,708	15,915
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases	0	0
Sales and settlements		(11,529)
Transfers to REO	0	0
VIE consolidation		0
VIE deconsolidation		0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	1,528
Unrealized gains/(losses), net on assets	(206)	(2,609)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	(330)	(1,597)
Ending balance	1,172	1,708	15,915
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Subtotal Agency MBS | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	(206)	(616)
Level III | Fair Value, Measurements, Recurring | Non-Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	34,369	45,814
Transfers into Level III from Level II	5,683	0
Transfers from Level III into Level II	(32,471)	0
Purchases	0	0
Sales and settlements		(12,658)
Transfers to REO	0	0
VIE consolidation		0
VIE deconsolidation		6,852
Loan modifications / capitalized interest	0	0
Principal repayments	(256)	(710)
Realized gains/(losses), net on assets	0	(60)
Unrealized gains/(losses), net on assets	698	(4,013)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	(178)	(856)
Ending balance	7,845	34,369	45,814
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Non-Agency MBS | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	(1,173)	(3,783)
Level III | Fair Value, Measurements, Recurring | Other securities
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	8,593	17,196
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	(10,306)	(6,482)
Purchases	0	0
Sales and settlements		0
Transfers to REO	0	0
VIE consolidation		0
VIE deconsolidation		0
Loan modifications / capitalized interest	0	0
Principal repayments	0	(154)
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	1,657	(1,949)
Unrealized gains (losses), net on liabilities	0	0
Premium and discount amortization, net	56	(18)
Ending balance	0	8,593	$ 17,196
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Other securities | Assets
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	$ 0	$ (599)