Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Assets
Derivative assets	$ 13,776	$ 14,466
Liabilities
Derivative liabilities	6,370	10,130
Securitized debt	1,057
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	(8,574)	(11,180)	$ (22,873)
Premium and discount amortization, net	(7,398)	(5,374)	2,070
Residential Whole Loans
Assets
Fair value	1,200,566	1,041,885
Residential Bridge Loans
Assets
Fair value	33,269	211,999
Commercial Loans
Assets
Fair value	370,213	216,123
Securitized Commercial Loans
Assets
Fair value	909,040
Agency CMBS
Assets
Estimated Fair Value	1,435,477	1,481,984
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(2,327)	(646)	507
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	3,092	4,158
Agency RMBS
Assets
Estimated Fair Value	340,771
Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	10,343	12,135
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,572	7,702
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,572	7,702
Subtotal Agency MBS
Assets
Estimated Fair Value	1,795,255	1,505,979
Non-Agency RMBS
Assets
Estimated Fair Value	38,131	39,026
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	(1,331)	(996)	0
Premium and discount amortization, net	(2,214)	(1,073)	525
Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	7,683	11,529
Non-Agency CMBS
Assets
Estimated Fair Value	316,019	200,301
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	(6,565)	(8,660)	(15,117)
Premium and discount amortization, net	4,017	6,366	8,276
Subtotal Non-Agency MBS
Assets
Estimated Fair Value	361,833	250,856
Other securities
Assets
Estimated Fair Value	80,161	59,906
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	(604)	(717)	(1,982)
Premium and discount amortization, net	(6,472)	(6,371)	1,559
Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	2,237,249	1,816,741
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net	(10,049)	(6,816)	(2,191)
Non-Agency MBS
Assets
Estimated Fair Value	361,833	250,856
Level I
Assets
Derivative assets	0	0
Total	0	0
Liabilities
Derivative liabilities	0	4,657
Securitized debt	0	0
Total Liabilities	0	4,657
Level I | Residential Whole Loans
Assets
Fair value	0	0
Level I | Residential Bridge Loans
Assets
Fair value	0
Level I | Commercial Loans
Assets
Fair value	0	0
Level I | Securitized Commercial Loans
Assets
Fair value	0	0
Level I | Agency CMBS
Assets
Estimated Fair Value	0	0
Level I | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0	0
Level I | Agency RMBS
Assets
Estimated Fair Value	0
Level I | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0	0
Level I | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value		0
Level I | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0
Level I | Subtotal Agency MBS
Assets
Estimated Fair Value	0	0
Level I | Non-Agency RMBS
Assets
Estimated Fair Value	0	0
Level I | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0	0
Level I | Non-Agency CMBS
Assets
Estimated Fair Value	0	0
Level I | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	0	0
Level I | Other securities
Assets
Estimated Fair Value	0	0
Level I | Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	0	0
Level II
Assets
Derivative assets	5,111	2,606
Total	2,163,435	1,740,004
Liabilities
Derivative liabilities	6,370	5,473
Securitized debt	680,586	947,340
Total Liabilities	686,956	952,813
Level II | Residential Whole Loans
Assets
Fair value	0	0
Level II | Residential Bridge Loans
Assets
Fair value	0	0
Level II | Commercial Loans
Assets
Fair value	0	0
Level II | Securitized Commercial Loans
Assets
Fair value	0	0
Level II | Agency CMBS
Assets
Estimated Fair Value	1,435,477	1,481,984
Level II | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	3,092	4,158
Level II | Agency RMBS
Assets
Estimated Fair Value	340,771
Level II | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0	0
Level II | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value		0
Level II | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0
Level II | Subtotal Agency MBS
Assets
Estimated Fair Value	1,779,340	1,486,142
Level II | Non-Agency RMBS
Assets
Estimated Fair Value	0	0
Level II | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0	0
Level II | Non-Agency CMBS
Assets
Estimated Fair Value	316,019	200,301
Level II | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	316,019	200,301
Level II | Other securities
Assets
Estimated Fair Value	62,965	50,955
Level II | Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	2,158,324	1,737,398
Level III
Assets
Derivative assets	0	0
Total	2,767,307	2,562,861
Liabilities
Derivative liabilities	0	0
Securitized debt		2,286
Total Liabilities	1,057	2,286
Level III | Residential Whole Loans
Assets
Fair value	1,375,860
Level III | Commercial Loans
Assets
Fair value		216,123
Level III | Securitized Commercial Loans
Assets
Fair value		1,013,511
Level III | Agency CMBS
Assets
Estimated Fair Value	0	0
Level III | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	0	0
Level III | Agency RMBS
Assets
Estimated Fair Value	0
Level III | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	10,343	12,135
Level III | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value		7,702
Level III | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,572
Level III | Subtotal Agency MBS
Assets
Estimated Fair Value	15,915	19,837
Level III | Non-Agency RMBS
Assets
Estimated Fair Value	38,131	39,026
Level III | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	7,683	11,529
Level III | Non-Agency CMBS
Assets
Estimated Fair Value	0	0
Level III | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	45,814	50,555
Level III | Other securities
Assets
Estimated Fair Value	17,196	8,951
Level III | Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	78,925	79,343
Recurring basis | Residential Bridge Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Realized gains/(losses), net on assets	(351)
Recurring basis | Commercial Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	216,123
Principal repayments	(121,245)
Ending balance		216,123
Recurring basis | Subtotal Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Sales and settlements	401
Recurring basis | Level III | Residential Whole Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,041,885	237,423
Purchases	544,426	860,576
Principal repayments	(228,163)	(55,186)
Unrealized gains/(losses), net	20,887	(415)
Premium and discount amortization, net	(3,175)	(513)
Ending balance	1,375,860	1,041,885	237,423
Recurring basis | Level III | Residential Bridge Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	211,999	64,526
Purchases		207,705
Transfers to REO	(2,677)
Principal repayments	(175,422)	(57,528)
Unrealized gains/(losses), net	397	(1,806)
Premium and discount amortization, net	(677)	(898)
Ending balance	33,269	211,999	64,526
Recurring basis | Level III | Commercial Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	216,123	0
Purchases	274,422	215,322
Unrealized gains/(losses), net	(122)	631
Premium and discount amortization, net	1,035	170
Ending balance	370,213	216,123	0
Recurring basis | Level III | Securitized Commercial Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	1,013,511	24,876
Purchases	1,113,231	1,353,020
Principal repayments	(1,214,688)	(361,782)
Unrealized gains/(losses), net	(1,070)	(16)
Premium and discount amortization, net	(1,944)	(2,587)
Ending balance	909,040	1,013,511	24,876
Recurring basis | Level III | Subtotal Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	19,837	17,217
Transfers into Level III from Level II		22,795
Transfers from Level III into Level II		(16,805)
Purchases		2,093
Principal repayments		(53)
Other than temporary impairment	(222)	(735)
Unrealized gains/(losses), net	762	(630)
Premium and discount amortization, net	(4,061)	(4,045)
Ending balance	15,915	19,837	17,217
Recurring basis | Level III | Other securities
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	8,951	9,239
Transfers into Level III from Level II	8,386	9,708
Transfers from Level III into Level II		(8,697)
Principal repayments	(555)	(604)
Other than temporary impairment		(161)
Unrealized gains/(losses), net	693	(532)
Premium and discount amortization, net	(279)	(2)
Ending balance	17,196	8,951	9,239
Recurring basis | Level III | Non-Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	50,555	8,735
Transfers into Level III from Level II		39,084
Purchases		8,602
Sales and settlements		(4,180)
Principal repayments	(965)	(307)
Realized gains/(losses), net on assets		258
Other than temporary impairment	(1,332)	(918)
Unrealized gains/(losses), net	(229)	1,183
Premium and discount amortization, net	(2,215)	(1,902)
Ending balance	45,814	50,555	8,735
Securitized Debt | Recurring basis | Level III
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	375	(1,998)
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance	2,286	10,945
Transfers from Level III into Level II		(10,899)
Sales and settlements		12
Principal repayments		(44)
Unrealized gains (losses), net	(2,373)	1,996
Premium and discount amortization, net	(2,625)	276
Ending balance	1,057	2,286	$ 10,945
Securitized Debt | Recurring basis | Level III | Securitized Debt
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Sales and settlements	3,769
Assets | Recurring basis | Level III | Residential Whole Loans
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	21,768	351
Assets | Recurring basis | Level III | Residential Bridge Loans
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	(488)	(1,370)
Assets | Recurring basis | Level III | Commercial Loans
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	128	631
Assets | Recurring basis | Level III | Securitized Commercial Loans
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	(1,042)	(16)
Assets | Recurring basis | Level III | Subtotal Agency MBS
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	780	(272)
Assets | Recurring basis | Level III | Other securities
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	693	(464)
Assets | Recurring basis | Level III | Non-Agency MBS
Fair Value of Financial Instruments
Debt Securities, Available-for-sale, Unrealized Gain (Loss)	(229)	1,184
Estimated Fair Value
Assets
Derivative assets	5,111	2,606
Total	4,930,742	4,302,865
Liabilities
Derivative liabilities	6,370	10,130
Securitized debt	681,643	949,626
Total Liabilities	688,013	959,756
Estimated Fair Value | Residential Whole Loans
Assets
Fair value	1,375,860	1,041,885
Estimated Fair Value | Residential Bridge Loans
Assets
Fair value	33,269	211,999
Estimated Fair Value | Commercial Loans
Assets
Fair value	370,213	216,123
Estimated Fair Value | Securitized Commercial Loans
Assets
Fair value	909,040	1,013,511
Estimated Fair Value | Agency CMBS
Assets
Estimated Fair Value	1,435,477	1,481,984
Estimated Fair Value | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	3,092	4,158
Estimated Fair Value | Agency RMBS
Assets
Estimated Fair Value	340,771
Estimated Fair Value | Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	10,343	12,135
Estimated Fair Value | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value		7,702
Estimated Fair Value | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,572
Estimated Fair Value | Subtotal Agency MBS
Assets
Estimated Fair Value	1,795,255	1,505,979
Estimated Fair Value | Non-Agency RMBS
Assets
Estimated Fair Value	38,131	39,026
Estimated Fair Value | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	7,683	11,529
Estimated Fair Value | Non-Agency CMBS
Assets
Estimated Fair Value	316,019	200,301
Estimated Fair Value | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	361,833	250,856
Estimated Fair Value | Other securities
Assets
Estimated Fair Value	80,161	59,906
Estimated Fair Value | Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	2,237,249	1,816,741
Estimated Fair Value | Recurring basis
Liabilities
Securitized debt	810,914	0
Total Liabilities	$ 3,849,179	$ 2,932,146