Stockholders' Equity - Components of AOCI (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Accumulated other comprehensive income (loss) from derivative instruments:
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	$ 83,965	$ (210,424)	$ (9,885)
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	9,072	193,162	1,508
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(23,729)	(25,839)	(25,544)
Currency translation adjustments on investment in unconsolidated venture	(1,158)	(447)	863
Total other comprehensive income/(loss), net	68,150	(43,548)	(33,058)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	2,286,697	2,656,878	2,270,184
Total other comprehensive income/(loss), net	68,150	(43,548)	(33,058)
Other comprehensive income/(loss) attributable to non-controlling interest		1,233
Rebalancing of ownership percentage of non-controlling interest		(1)
Purchase of OP units from non-controlling interest		(21,764)
Ending balance	2,931,899	2,286,697	2,656,878
Equity method investments, including portion attributable to noncontrolling interest
Accumulated other comprehensive income (loss) from derivative instruments:
Currency translation adjustments on investment in unconsolidated venture	(1,158)	(447)
Total other comprehensive income/(loss), net	(1,158)	(447)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss), net	(1,158)	(447)
Other comprehensive income/(loss) attributable to non-controlling interest		6
Purchase of OP units from non-controlling interest		7
Available-for-sale securities, including portion attributable to noncontrolling interest
Accumulated other comprehensive income (loss) from derivative instruments:
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	83,965	(210,424)
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	9,072	193,162
Total other comprehensive income/(loss), net	93,037	(17,262)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss), net	93,037	(17,262)
Other comprehensive income/(loss) attributable to non-controlling interest		927
Rebalancing of ownership percentage of non-controlling interest		(1)
Purchase of OP units from non-controlling interest		812
Derivatives and hedging, including portion attributable to noncontrolling interest
Accumulated other comprehensive income (loss) from derivative instruments:
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(23,729)	(25,839)
Total other comprehensive income/(loss), net	(23,729)	(25,839)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss), net	(23,729)	(25,839)
Other comprehensive income/(loss) attributable to non-controlling interest		300
Purchase of OP units from non-controlling interest		1,281
Equity method investment attributable to parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	513	947
Ending balance	(645)	513	947
Available-for-sale securities attributable to parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	120,664	136,188
Ending balance	213,701	120,664	136,188
Derivatives and hedging attributable to parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	99,636	123,894
Ending balance	75,907	99,636	123,894
AOCI attributable to parent
Accumulated other comprehensive income (loss) from derivative instruments:
Total other comprehensive income/(loss), net	68,150	(42,315)	(32,640)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	220,813	261,029	293,668
Total other comprehensive income/(loss), net	68,150	(42,315)	(32,640)
Rebalancing of ownership percentage of non-controlling interest		(1)	1
Purchase of OP units from non-controlling interest		2,100
Ending balance	$ 288,963	$ 220,813	$ 261,029