Stockholders' Equity - Components of Accumulated Other Comprehensive Income (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2018	Mar. 31, 2017
Accumulated Other Comprehensive Income (Loss) [Line Items]
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	$ (132,317)	$ 16,289
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	9,237	850
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(6,539)	(6,298)
Currency translation adjustments on investment in unconsolidated venture	312	(615)
Total other comprehensive income (loss)	(129,307)	10,226
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Balance at beginning of period	2,656,878
Total other comprehensive income/(loss)	(129,307)	10,226
Other comprehensive income/(loss) attributable to non-controlling interest	1,630	(129)
Balance at end of period	2,522,356
Equity method investments including portion attributable to noncontrolling interest
Accumulated Other Comprehensive Income (Loss) [Line Items]
Currency translation adjustments on investment in unconsolidated venture	312	(615)
Total other comprehensive income (loss)	312	(615)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss)	312	(615)
Other comprehensive income/(loss) attributable to non-controlling interest	(4)	8
Available-for-sale securities including portion attributable to noncontrolling interest
Accumulated Other Comprehensive Income (Loss) [Line Items]
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	(132,317)	16,289
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	9,237	850
Total other comprehensive income (loss)	(123,080)	17,139
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss)	(123,080)	17,139
Other comprehensive income/(loss) attributable to non-controlling interest	1,552	(216)
Derivative and hedging including portion attributable to noncontrolling interest
Accumulated Other Comprehensive Income (Loss) [Line Items]
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(6,539)	(6,298)
Total other comprehensive income (loss)	(6,539)	(6,298)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Total other comprehensive income/(loss)	(6,539)	(6,298)
Other comprehensive income/(loss) attributable to non-controlling interest	82	79
Equity method investments attributable to Parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Balance at beginning of period	947	95
Balance at end of period	1,255	(512)
Available-for-sale securities attributable to Parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Balance at beginning of period	136,188	144,458
Balance at end of period	14,660	161,381
Derivative and hedging attributable to Parent
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Balance at beginning of period	123,894	149,115
Balance at end of period	117,437	142,896
Accumulated other comprehensive income
Accumulated Other Comprehensive Income (Loss) [Line Items]
Total other comprehensive income (loss)	(127,677)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Balance at beginning of period	261,029	293,668
Total other comprehensive income/(loss)	(127,677)
Balance at end of period	$ 133,352	$ 303,765