Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019
Assets
Derivative assets	$ 33,675		$ 5,111
Liabilities
Derivative liabilities	43,967		6,370
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	0	$ (1,232)
Premium and discount amortization, net	(1,837)	(357)
Residential  Whole Loans
Assets
Fair value	1,309,795		1,375,860
Commercial Loans
Assets
Fair value	320,308
Agency CMBS
Assets
Estimated Fair Value	413,394		1,435,477
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(588)	(56)
Agency CMBS Interest-Only Strips, accounted for as derivatives
Assets
Estimated Fair Value	2,792		3,092
Subtotal Agency
Assets
Estimated Fair Value			340,771
Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	9,952		10,343
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	4,490		5,572
Agency MBS
Assets
Estimated Fair Value	430,628		1,795,255
Non-Agency CMBS
Assets
Estimated Fair Value	250,309		316,019
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment		(241)
Premium and discount amortization, net	(641)	(477)
Non-Agency RMBS
Assets
Estimated Fair Value	21,123		38,131
Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	5,174		7,683
Subtotal Non-Agency MBS
Assets
Estimated Fair Value	276,606		361,833
Other securities
Assets
Estimated Fair Value	47,411		80,161
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(1,464)	(1,591)
Total
Assets
Estimated Fair Value	754,645		2,237,249
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(2,567)	(1,855)
Non-Agency MBS
Assets
Estimated Fair Value	276,606		361,833
Fair Value, Measurements, Recurring
Assets
Derivative assets	33,675		5,111
Total Assets	2,921,604		4,930,742
Liabilities
Derivative liabilities	43,967		6,370
Securitized debt, at fair value			681,643
Total Liabilities	440,791		688,013
Fair Value, Measurements, Recurring | Securitized debt
Liabilities
Securitized debt, at fair value	396,824
Fair Value, Measurements, Recurring | Residential  Whole Loans
Assets
Fair value	1,309,795		1,375,860
Fair Value, Measurements, Recurring | Residential Bridge Loans
Assets
Fair value	26,050		33,269
Fair Value, Measurements, Recurring | Commercial Loans
Assets
Fair value	320,308		909,040
Fair Value, Measurements, Recurring | Securitized commercial loans
Assets
Fair value	477,131		370,213
Fair Value, Measurements, Recurring | Agency CMBS
Assets
Estimated Fair Value	413,394		1,435,477
Fair Value, Measurements, Recurring | Agency CMBS Interest-Only Strips, accounted for as derivatives
Assets
Estimated Fair Value	2,792		3,092
Fair Value, Measurements, Recurring | Subtotal Agency
Assets
Estimated Fair Value			340,771
Fair Value, Measurements, Recurring | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	9,952		10,343
Fair Value, Measurements, Recurring | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	4,490		5,572
Fair Value, Measurements, Recurring | Agency MBS
Assets
Estimated Fair Value	430,628		1,795,255
Fair Value, Measurements, Recurring | Non-Agency CMBS
Assets
Estimated Fair Value	250,309		316,019
Fair Value, Measurements, Recurring | Non-Agency RMBS
Assets
Estimated Fair Value	21,123		38,131
Fair Value, Measurements, Recurring | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	5,174		7,683
Fair Value, Measurements, Recurring | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	276,606		361,833
Fair Value, Measurements, Recurring | Other securities
Assets
Estimated Fair Value	47,411		80,161
Fair Value, Measurements, Recurring | Total
Assets
Estimated Fair Value	754,645		2,237,249
Fair Value, Measurements, Recurring | Level I
Assets
Derivative assets	0		0
Total Assets	0		0
Liabilities
Derivative liabilities	0		0
Securitized debt, at fair value			0
Total Liabilities	0		0
Fair Value, Measurements, Recurring | Level I | Securitized debt
Liabilities
Securitized debt, at fair value	0
Fair Value, Measurements, Recurring | Level I | Residential  Whole Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level I | Residential Bridge Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level I | Commercial Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level I | Securitized commercial loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level I | Agency CMBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Agency CMBS Interest-Only Strips, accounted for as derivatives
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Subtotal Agency
Assets
Estimated Fair Value			0
Fair Value, Measurements, Recurring | Level I | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Agency MBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Non-Agency CMBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Non-Agency RMBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Other securities
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level I | Total
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level II
Assets
Derivative assets	33,675		5,111
Total Assets	733,345		2,163,435
Liabilities
Derivative liabilities	43,967		6,370
Securitized debt, at fair value			680,586
Total Liabilities	440,630		686,956
Fair Value, Measurements, Recurring | Level II | Securitized debt
Liabilities
Securitized debt, at fair value	396,663
Fair Value, Measurements, Recurring | Level II | Residential  Whole Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level II | Residential Bridge Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level II | Commercial Loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level II | Securitized commercial loans
Assets
Fair value	0		0
Fair Value, Measurements, Recurring | Level II | Agency CMBS
Assets
Estimated Fair Value	413,394		1,435,477
Fair Value, Measurements, Recurring | Level II | Agency CMBS Interest-Only Strips, accounted for as derivatives
Assets
Estimated Fair Value	2,792		3,092
Fair Value, Measurements, Recurring | Level II | Subtotal Agency
Assets
Estimated Fair Value			340,771
Fair Value, Measurements, Recurring | Level II | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level II | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level II | Agency MBS
Assets
Estimated Fair Value	416,186		1,779,340
Fair Value, Measurements, Recurring | Level II | Non-Agency CMBS
Assets
Estimated Fair Value	243,444		316,019
Fair Value, Measurements, Recurring | Level II | Non-Agency RMBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level II | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level II | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	243,444		316,019
Fair Value, Measurements, Recurring | Level II | Other securities
Assets
Estimated Fair Value	40,040		62,965
Fair Value, Measurements, Recurring | Level II | Total
Assets
Estimated Fair Value	699,670		2,158,324
Fair Value, Measurements, Recurring | Level III
Assets
Derivative assets	0		0
Total Assets	2,188,259		2,767,307
Liabilities
Derivative liabilities	0		0
Securitized debt, at fair value			1,057
Total Liabilities	161		1,057
Fair Value, Measurements, Recurring | Level III | Securitized debt
Liabilities
Securitized debt, at fair value	161
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(519)	(316)
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,057	2,286
Sales and settlements		(3,769)
Unrealized (gains)/losses, net on liabilities	(377)	(1,970)
Ending balance	161	3,769
Unrealized gains/(losses), net on assets held at the end of the period	377	(28)
Fair Value, Measurements, Recurring | Level III | Residential  Whole Loans
Assets
Fair value	1,309,795		1,375,860
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,375,860	1,041,885
Purchases	111,486	248,105
Principal repayments	(80,361)	(28,532)
Unrealized gains/(losses), net on assets	(96,160)	5,886
Premium and discount amortization, net	(1,030)	(181)
Ending balance	1,309,795	1,267,163
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments	(80,361)	(28,532)
Unrealized gains/(losses), net on assets held at the end of the period	(94,347)	6,108
Fair Value, Measurements, Recurring | Level III | Residential Bridge Loans
Assets
Fair value	26,050		33,269
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	33,269	211,999
Purchases		0
Transfers to REO	(489)
Principal repayments	(6,408)	(66,612)
Realized gains/(losses), net on assets	(85)	(87)
Unrealized gains/(losses), net on assets	(218)	(780)
Premium and discount amortization, net	(19)	(350)
Ending balance	26,050	144,170
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments	(6,408)	(66,612)
Unrealized gains/(losses), net on assets held at the end of the period	(417)	(780)
Fair Value, Measurements, Recurring | Level III | Commercial Loans
Assets
Fair value	320,308		909,040
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	370,213	216,123
Purchases		121,189
Principal repayments	(37,638)	(165)
Unrealized gains/(losses), net on assets	(12,462)	223
Premium and discount amortization, net	195	208
Ending balance	320,308	337,578
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments	(37,638)	(165)
Unrealized gains/(losses), net on assets held at the end of the period	(12,460)	223
Fair Value, Measurements, Recurring | Level III | Securitized commercial loans
Assets
Fair value	477,131		370,213
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	909,040	1,013,511
Purchases		903,770
VIE deconsolidation	(150,804)
Principal repayments	(154,701)	(988,714)
Unrealized gains/(losses), net on assets	(127,171)	1,349
Premium and discount amortization, net	767	(442)
Ending balance	477,131	929,474
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments	(154,701)	(988,714)
Unrealized gains/(losses), net on assets held at the end of the period	(68,013)	1,377
Fair Value, Measurements, Recurring | Level III | Agency CMBS
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level III | Agency CMBS Interest-Only Strips, accounted for as derivatives
Assets
Estimated Fair Value	0		0
Fair Value, Measurements, Recurring | Level III | Subtotal Agency
Assets
Estimated Fair Value			0
Fair Value, Measurements, Recurring | Level III | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	9,952		10,343
Fair Value, Measurements, Recurring | Level III | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	4,490		5,572
Fair Value, Measurements, Recurring | Level III | Agency MBS
Assets
Estimated Fair Value	14,442		15,915
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	15,915	19,837
Transfers into Level III from Level II		0
Transfers from Level III into Level II		0
Purchases		0
Principal repayments		0
Other than temporary impairment		(25)
Unrealized gains/(losses), net on assets	(534)	387
Premium and discount amortization, net	(939)	(1,091)
Ending balance	14,442	19,108
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments		0
Unrealized gains/(losses), net on assets held at the end of the period	(534)	387
Fair Value, Measurements, Recurring | Level III | Non-Agency CMBS
Assets
Estimated Fair Value	6,865		0
Fair Value, Measurements, Recurring | Level III | Non-Agency RMBS
Assets
Estimated Fair Value	21,123		38,131
Fair Value, Measurements, Recurring | Level III | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	5,174		7,683
Fair Value, Measurements, Recurring | Level III | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	33,162		45,814
Fair Value, Measurements, Recurring | Level III | Other securities
Assets
Estimated Fair Value	7,371		17,196
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	17,196	8,951
Transfers into Level III from Level II		8,386
Transfers from Level III into Level II	(6,482)	0
Principal repayments	(153)	0
Unrealized gains/(losses), net on assets	(3,120)	121
Premium and discount amortization, net	(70)	(78)
Ending balance	7,371	17,380
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments	(153)	0
Unrealized gains/(losses), net on assets held at the end of the period	(1,770)	121
Fair Value, Measurements, Recurring | Level III | Total
Assets
Estimated Fair Value	54,975		$ 78,925
Fair Value, Measurements, Recurring | Level III | Non-Agency MBS
Investment assets measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	45,814	50,555
Transfers into Level III from Level II		0
VIE deconsolidation	6,852
Principal repayments	(320)	(252)
Realized gains/(losses), net on assets	16
Other than temporary impairment		(241)
Unrealized gains/(losses), net on assets	(5,835)	(1,193)
Premium and discount amortization, net	(631)	(477)
Ending balance	33,162	48,392
Investment liabilities measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Sales and settlements	(12,702)
Principal repayments	(320)	(252)
Unrealized gains/(losses), net on assets held at the end of the period	$ (5,605)	$ (1,193)