Investment In Securities - Carrying Value of Trading Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount		$ 975,292
Amortized Cost	$ 824,844	855,732
Gross Unrealized Loss		(70)
Gross Unrealized Gain		83,522
Fair Value	883,601	939,184
Credit Risk Transfer
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount	754,729	661,181
Amortized Cost	751,940	653,681
Gross Unrealized Loss	0	0
Gross Unrealized Gain	52,024	76,303
Fair Value	803,964	729,984
Non-Agency Securities
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount	93,723	105,942
Amortized Cost	72,904	83,150
Gross Unrealized Loss	(3)	(13)
Gross Unrealized Gain	6,736	6,794
Fair Value	79,637	89,931
Credit Risk and Non-Agency Securities
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount	848,452	767,123
Amortized Cost	824,844	736,831
Gross Unrealized Loss	(3)	(13)
Gross Unrealized Gain	58,760	83,097
Fair Value	883,601	819,915	$ 975,830
U.S. Treasury Securities
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount		100,000
Amortized Cost		98,703
Gross Unrealized Loss		(57)
Gross Unrealized Gain		0
Fair Value	0	98,646	0
Interest-Only Securities
Debt and Equity Securities, FV-NI [Line Items]
Principal Amount		108,169
Amortized Cost		20,198
Gross Unrealized Loss		0
Gross Unrealized Gain		425
Fair Value	$ 0	$ 20,623	$ 25,752