Fair Value of Financial Instruments - Financial Instruments Measured at Level III (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	$ (260)	$ (2,228)
Fair value asset recurring basis still held unrealized gain loss statement of income extensible list not disclosed flag	Unrealized gains/(losses), net on assets(1)
Level III | Fair Value, Measurements, Recurring | Securitized debt
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	$ 8,927	14,919
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	0	0
Unrealized (gains)/losses, net on liabilities	(97)	811
Premium and discount amortization, net	(858)	(811)
Ending balance	7,972	14,919
Unrealized gain (loss)	97	(811)
Level III | Fair Value, Measurements, Recurring | Securitized debt | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	0
Level III | Fair Value, Measurements, Recurring | Residential  Whole Loans
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,091,145	1,023,502
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		118,738
Sales and settlements	0
Loan modifications / capitalized interest	7	64
Principal repayments	(30,514)	(95,224)
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	14,488	(42,327)
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	(709)	(2,043)
Ending balance	1,074,417	1,002,710
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Residential  Whole Loans | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	13,183	(40,637)
Level III | Fair Value, Measurements, Recurring | Residential Bridge Loans
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	2,849	5,428
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	0
Loan modifications / capitalized interest	0	0
Principal repayments	(75)	(105)
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	8	27
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	0	0
Ending balance	2,782	5,350
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Residential Bridge Loans | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	0	25
Level III | Fair Value, Measurements, Recurring | Commercial Loans
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	90,002	130,572
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	(8,776)
Loan modifications / capitalized interest	0	0
Principal repayments	(930)	(4)
Realized gains/(losses), net on assets	(81,223)	0
Unrealized gains/(losses), net on assets	80,055	(2,073)
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	54	0
Ending balance	79,182	128,495
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Commercial Loans | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	(1,168)	(2,073)
Level III | Fair Value, Measurements, Recurring | Securitized Commercial Loans
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,085,103	1,355,808
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	(4,036)	(73,564)
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	7,157	6,699
Ending balance	1,088,224	1,288,943
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Securitized Commercial Loans | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	(4,036)	(73,564)
Level III | Fair Value, Measurements, Recurring | Agency MBS
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	767	1,172
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	66	(157)
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	4	(75)
Ending balance	837	940
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Agency MBS | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	66	(157)
Level III | Fair Value, Measurements, Recurring | Non-Agency MBS
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,204	7,845
Transfers into Level III from Level II	0	0
Transfers from Level III into Level II	0	0
Purchases		0
Sales and settlements	0
Loan modifications / capitalized interest	0	0
Principal repayments	0	0
Realized gains/(losses), net on assets	0	0
Unrealized gains/(losses), net on assets	490	(1,104)
Unrealized (gains)/losses, net on liabilities	0	0
Premium and discount amortization, net	(104)	(82)
Ending balance	1,590	6,659
Unrealized gain (loss)	0	0
Level III | Fair Value, Measurements, Recurring | Non-Agency MBS | Assets
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Unrealized gain (loss)	$ 490	$ (1,104)