Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Fair Value of Financial Instruments
Derivative credit risk valuation adjustment, derivative assets					$ 0					$ 0
Assets
Derivative assets	$ 728				20,571				$ 728	20,571
Liabilities
Derivative liabilities	4,346				182,158				4,346	182,158
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment	(2,972)	$ (7,225)	$ (6,579)	$ (6,097)	(10,155)	$ (4,978)	$ (6,356)	$ (10,797)	(22,873)	(32,286)	$ (19,791)
Premium and discount amortization, net									2,070	548	(5,086)
Residential Whole-Loans
Assets
Fair value	237,423				192,136				237,423	192,136
Securitized commercial loan
Assets
Fair value	24,876				24,225				24,876	24,225
Agency RMBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment									(5,774)	(6,090)	(4,005)
Premium and discount amortization, net									(13,058)	(24,836)	(48,119)
20 Year Mortgage
Assets
Estimated Fair Value	53,783				498,470				53,783	498,470
30 Year Mortgage
Assets
Estimated Fair Value	241,642				935,207				241,642	935,207
40 Years Mortgage
Assets
Estimated Fair Value	376,752								376,752
Agency RMBS Interest Only Strips
Assets
Estimated Fair Value	15,437				19,790				15,437	19,790
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	10,419				16,503				10,419	16,503
Agency CMBS
Assets
Estimated Fair Value	2,154,800				363,664				2,154,800	363,664
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net									507	(1,701)	(2,102)
Agency CMBS Interest-Only Strips
Assets
Estimated Fair Value	10				231				10	231
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,757				7,729				5,757	7,729
Subtotal Agency MBS
Assets
Estimated Fair Value	2,858,600				1,841,594				2,858,600	1,841,594
Non-Agency Mortgage-Backed Securities [Member]
Assets
Estimated Fair Value	378,158				667,161				378,158	667,161
Non-Agency RMBS
Assets
Estimated Fair Value	90,832				241,041				90,832	241,041
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment									0	(9,511)	(9,216)
Premium and discount amortization, net									525	(5,243)	(8,008)
Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	8,722				64,116				8,722	64,116
Non-Agency CMBS
Assets
Estimated Fair Value	278,604				358,919				278,604	358,919
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment									(15,117)	(13,025)	(4,061)
Premium and discount amortization, net									8,276	7,431	3,687
Subtotal Non-Agency MBS
Assets
Estimated Fair Value	378,158				667,161				378,158	667,161
Other securities
Assets
Estimated Fair Value	122,065				67,762				122,065	67,762
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Other than temporary impairment									(1,982)	(3,660)	(2,509)
Premium and discount amortization, net									1,559	2,977	2,648
Mortgage-backed securities and other securities
Assets
Estimated Fair Value	3,358,823				2,576,517				3,358,823	2,576,517
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Premium and discount amortization, net									(2,191)	(21,372)	(51,894)
Level I
Assets
Derivative assets	728				71				728	71
Total	728				71				728	71
Liabilities
Derivative liabilities	50				2,487				50	2,487
Total Liabilities	50				2,487				50	2,487
Level II
Assets
Derivative assets	0				20,500				0	20,500
Total	3,323,632				2,417,026				3,323,632	2,417,026
Liabilities
Derivative liabilities	4,296				177,998				4,296	177,998
Total Liabilities	4,296				177,998				4,296	177,998
Level II | 20 Year Mortgage
Assets
Estimated Fair Value	53,783				498,470				53,783	498,470
Level II | 30 Year Mortgage
Assets
Estimated Fair Value	241,642				935,207				241,642	935,207
Level II | 40 Years Mortgage
Assets
Estimated Fair Value	376,752								376,752
Level II | Agency RMBS Interest Only Strips
Assets
Estimated Fair Value	15,437				19,790				15,437	19,790
Level II | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	10,419				16,503				10,419	16,503
Level II | Agency CMBS
Assets
Estimated Fair Value	2,137,583				290,605				2,137,583	290,605
Level II | Agency CMBS Interest-Only Strips
Assets
Estimated Fair Value	10				231				10	231
Level II | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,757				7,729				5,757	7,729
Level II | Subtotal Agency MBS
Assets
Estimated Fair Value	2,841,383				1,768,535				2,841,383	1,768,535
Level II | Non-Agency RMBS
Assets
Estimated Fair Value	90,819				240,422				90,819	240,422
Level II | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	0								0
Level II | Non-Agency CMBS
Assets
Estimated Fair Value	278,604				351,163				278,604	351,163
Level II | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	369,423				591,585				369,423	591,585
Level II | Other securities
Assets
Estimated Fair Value	112,826				36,406				112,826	36,406
Level II | Mortgage-backed securities and other securities
Assets
Estimated Fair Value	3,323,632				2,396,526				3,323,632	2,396,526
Level III
Assets
Derivative assets	0				0				0	0
Total	362,016				396,352				362,016	396,352
Liabilities
Derivative liabilities	0				1,673				0	1,673
Securitized debt	10,945				10,659				10,945	10,659
Total Liabilities	10,945				12,332				10,945	12,332
Level III | Residential Whole-Loans
Assets
Fair value	237,423				192,136				237,423	192,136
Level III | Residential Bridge Loans
Assets
Fair value	64,526								64,526
Level III | Securitized commercial loan
Assets
Fair value	24,876				24,225				24,876	24,225
Level III | Agency CMBS
Assets
Estimated Fair Value	17,217				73,059				17,217	73,059
Level III | Subtotal Agency MBS
Assets
Estimated Fair Value	17,217				73,059				17,217	73,059
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				73,059					73,059
Ending balance	17,217				73,059				17,217	73,059
Level III | Non-Agency Mortgage-Backed Securities [Member]
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				75,576				369,650	75,576	369,650
Ending balance	8,735				75,576				8,735	75,576	369,650
Level III | Non-Agency RMBS
Assets
Estimated Fair Value	13				619				13	619
Level III | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	8,722				64,116				8,722	64,116
Level III | Non-Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value					3,085					3,085
Level III | Non-Agency CMBS
Assets
Estimated Fair Value	0				7,756				0	7,756
Level III | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	8,735				75,576				8,735	75,576
Level III | Other securities
Assets
Estimated Fair Value	9,239				31,356				9,239	31,356
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				31,356				71,996	31,356	71,996
Ending balance	9,239				31,356				9,239	31,356	71,996
Level III | Mortgage-backed securities and other securities
Assets
Estimated Fair Value	35,191				179,991				35,191	179,991
Level III | Recurring basis | Securitized debt
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Principal repayments									(68)
Unrealized gains (losses), net									354	(341)
Gross unrealized gains									0	341
Gross unrealized losses									(354)	0
Beginning balance				10,659				11,000	10,659	11,000
Ending balance	10,945				10,659				10,945	10,659	11,000
Level III | Recurring basis | Derivative Liability
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Sales and settlements										(38)
Sales and settlements									552
Realized gains/(losses), net										38
Realized gains/(losses), net									(552)
Unrealized gains (losses), net									(1,673)	1,673
Gross unrealized gains									0	0
Gross unrealized losses									0	(1,700)
Beginning balance				1,673				0	1,673	0
Ending balance	0				1,673				0	1,673	0
Level III | Recurring basis | Residential Whole-Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				192,136				218,538	192,136	218,538
Purchases									86,921	28,226
Principal repayments									(41,312)	(52,573)
Unrealized gains/(losses), net									724	(42)
Premium and discount amortization, net									(1,046)	(2,013)
Ending balance	237,423				192,136				237,423	192,136	218,538
Gross unrealized gains									1,800	1,200
Gross unrealized losses									(400)	(663)
Level III | Recurring basis | Residential Bridge Loans
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				0					0
Purchases									63,969
Unrealized gains/(losses), net									571
Premium and discount amortization, net									(14)
Ending balance	64,526				0				64,526	0
Gross unrealized gains									655
Gross unrealized losses									(84)
Level III | Recurring basis | Securitized commercial loan
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				24,225				25,000	24,225	25,000
Principal repayments									(154)
Unrealized gains/(losses), net									805	(775)
Ending balance	24,876				24,225				24,876	24,225	25,000
Gross unrealized gains									805	0
Gross unrealized losses									0	(775)
Level III | Recurring basis | Subtotal Agency MBS
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Beginning balance				$ 73,059				$ 24,690	73,059	24,690
Transfers out level III into Level II									(73,714)	(16,730)
Purchases									17,217	72,642
Sales and settlements										(6,776)
Principal repayments										(141)
Realized gains/(losses), net										(56)
Unrealized gains/(losses), net									635	(570)
Premium and discount amortization, net									20
Ending balance					73,059					73,059	$ 24,690
Gross unrealized gains									0	417
Gross unrealized losses									0	0
Level III | Recurring basis | Non-Agency Mortgage-Backed Securities [Member]
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Transfers out level III into Level II									(7,433)	(198,078)
Purchases									8,937	94
Sales and settlements									(60,132)	(49,560)
Principal repayments									(378)	(15,508)
Realized gains/(losses), net									2,623	(9,345)
Other than temporary impairment										3,436
Unrealized gains/(losses), net									(9,415)	(4,889)
Premium and discount amortization, net									(1,043)	(13,352)
Gross unrealized gains									0	492
Gross unrealized losses									(16)	(6,800)
Level III | Recurring basis | Other securities
Investments measured at fair value on recurring basis for which the entity has utilized Level III inputs to determine fair value
Transfers into Level III from Level II									9,470
Transfers out level III into Level II									(33,080)	(6,079)
Sales and settlements										(32,226)
Principal repayments									(425)	(2,049)
Realized gains/(losses), net										45
Other than temporary impairment									(1,823)	3,301
Unrealized gains/(losses), net									1,522	(86)
Premium and discount amortization, net									2,219	3,056
Gross unrealized gains									43	420
Gross unrealized losses									0	(823)
Estimated Fair Value
Assets
Derivative assets	728				20,571				728	20,571
Total	3,686,376				2,813,449				3,686,376	2,813,449
Liabilities
Derivative liabilities	4,346				182,158				4,346	182,158
Securitized debt	10,945				10,659				10,945	10,659
Total Liabilities	15,291				192,817				15,291	192,817
Estimated Fair Value | Residential Whole-Loans
Assets
Fair value	237,423				192,136				237,423	192,136
Estimated Fair Value | Residential Bridge Loans
Assets
Fair value	64,526								64,526
Estimated Fair Value | Securitized commercial loan
Assets
Fair value	24,876				24,225				24,876	24,225
Estimated Fair Value | 20 Year Mortgage
Assets
Estimated Fair Value	53,783				498,470				53,783	498,470
Estimated Fair Value | 30 Year Mortgage
Assets
Estimated Fair Value	241,642				935,207				241,642	935,207
Estimated Fair Value | 40 Years Mortgage
Assets
Estimated Fair Value	376,752								376,752
Estimated Fair Value | Agency RMBS Interest Only Strips
Assets
Estimated Fair Value	15,437				19,790				15,437	19,790
Estimated Fair Value | Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	10,419				16,503				10,419	16,503
Estimated Fair Value | Agency CMBS
Assets
Estimated Fair Value	2,154,800				363,664				2,154,800	363,664
Estimated Fair Value | Agency CMBS Interest-Only Strips
Assets
Estimated Fair Value	10				231				10	231
Estimated Fair Value | Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	5,757				7,729				5,757	7,729
Estimated Fair Value | Subtotal Agency MBS
Assets
Estimated Fair Value	2,858,600				1,841,594				2,858,600	1,841,594
Estimated Fair Value | Non-Agency RMBS
Assets
Estimated Fair Value	90,832				241,041				90,832	241,041
Estimated Fair Value | Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	8,722				64,116				8,722	64,116
Estimated Fair Value | Non-Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value					3,085					3,085
Estimated Fair Value | Non-Agency CMBS
Assets
Estimated Fair Value	278,604				358,919				278,604	358,919
Estimated Fair Value | Subtotal Non-Agency MBS
Assets
Estimated Fair Value	378,158				667,161				378,158	667,161
Estimated Fair Value | Other securities
Assets
Estimated Fair Value	122,065				67,762				122,065	67,762
Estimated Fair Value | Mortgage-backed securities and other securities
Assets
Estimated Fair Value	$ 3,358,823				$ 2,576,517				$ 3,358,823	$ 2,576,517