Stockholders' Equity - Components of Accumulated Other Comprehensive Income (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Accumulated other comprehensive income (loss) from derivative instruments:
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	$ 756	$ (223,416)	$ 83,965
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	0	13,940	9,072
Reclassification of unrealized loss on available-for-sale securities to (increase) decrease in provision for credit losses	0	1,768	0
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(22,000)	(23,794)	(23,729)
Currency translation adjustments on investment in unconsolidated venture	(75)	1,144	(1,158)
Total other comprehensive income (loss)	(21,319)	(230,358)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	1,367,158	2,931,899	2,286,697
Total other comprehensive income (loss)	(21,319)	(230,358)	68,150
Ending balance	1,402,135	1,367,158	2,931,899
Equity method investments
Accumulated other comprehensive income (loss) from derivative instruments:
Currency translation adjustments on investment in unconsolidated venture		1,144
Total other comprehensive income (loss)	(75)	1,144
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	499	(645)
Total other comprehensive income (loss)	(75)	1,144
Ending balance	424	499	(645)
Available-for-sale securities
Accumulated other comprehensive income (loss) from derivative instruments:
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	756	(223,416)
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net		13,940
Reclassification of unrealized loss on available-for-sale securities to (increase) decrease in provision for credit losses		1,768
Total other comprehensive income (loss)	756	(207,708)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	5,993	213,701
Total other comprehensive income (loss)	756	(207,708)
Ending balance	6,749	5,993	213,701
Derivatives and hedging
Accumulated other comprehensive income (loss) from derivative instruments:
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(22,000)	(23,794)
Total other comprehensive income (loss)	(22,000)	(23,794)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	52,113	75,907
Total other comprehensive income (loss)	(22,000)	(23,794)
Ending balance	30,113	52,113	75,907
Accumulated other comprehensive income
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	58,605	288,963	220,813
Total other comprehensive income (loss)	(21,319)	(230,358)	68,150
Ending balance	$ 37,286	$ 58,605	$ 288,963