Offsetting Assets and Liabilities- Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, gross amounts	$ 7,282	$ 8,665
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, gross amounts offset in the consolidated balance sheets	0	0
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, Net amounts of assets presented in the consolidated balance sheets	7,282	8,665
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, gross amounts not offset in the consolidated balance sheets, financial instruments	(7,282)	(8,665)
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, gross amounts not offset in the consolidated balance sheets, cash collateral	0	0
Agency and Non-Agency Interest-Only Strips, accounted for as derivatives included in MBS, net amount	0	0
Derivative asset, at fair value
Derivative asset, at fair value, gross amounts	33,675	5,111
Derivative asset, at fair value, gross amounts offset in the consolidated balance sheets	0	0
Derivative asset, at fair value, net amounts of assets presented in the consolidated balance sheets	33,675	5,111
Derivative asset, at fair value, gross amounts not offset in the consolidated balance sheets, financial instruments	(18,429)	(2,576)
Derivative asset, at fair value, gross amounts not offset in the consolidated balance sheets, cash collateral	(14,809)	0
Derivative asset, at fair value, net amount	437	2,535
Receivable under reverse repurchase agreements
Receivable under reverse repurchase agreements, gross amounts	24,826
Receivable under reverse repurchase agreements, gross amounts offset in the consolidated balance sheets	0
Receivable under reverse repurchase agreements, net amounts of assets presented in the consolidated balance sheets	24,826
Receivable under reverse repurchase agreements, gross amounts not offset in the consolidated balance sheets, financial instruments	(20,767)
Receivable under reverse repurchase agreements, gross amounts not offset in the consolidated balance sheets, cash collateral	0
Receivable under reverse repurchase agreements, net amount	4,059
Total assets
Total assets, gross amounts	65,783	13,776
Total assets, gross amounts offset in the consolidated balance sheets	0	0
Total assets, net amounts of assets presented in the consolidated balance sheets	65,783	13,776
Total assets, gross amount not offset in the consolidated balance sheets, financial instruments	(46,478)	(11,241)
Total assets, gross amount not offset in the consolidated balance sheets, cash collateral	(14,809)	0
Total assets, net amount	4,496	2,535
Derivative liability, at fair value
Derivative liability, at fair value, gross amount	43,967	6,370
Derivative liability, at fair value, gross amounts offset in the consolidated balance sheets	0	0
Derivative liability, at fair value, net amounts of assets presented in the consolidated balance sheets	43,967	6,370
Derivative liability, at fair value, gross amounts not offset in the consolidated balance sheets, financial instruments	(18,429)	(2,576)
Derivative liability, at fair value, gross amounts not offset in the consolidated balance sheets, cash collateral	(25,538)	(2,819)
Derivative liability, at fair value, net amount	0	975
Repurchase Agreements
Repurchase agreements, gross amount	1,553,715	2,824,801
Repurchase agreements, gross amounts offset in the consolidated balance sheets	0	0
Repurchase agreements, net amounts of assets presented in the consolidated balance sheets	1,553,715	2,824,801
Repurchase agreements, gross amounts not offset in the consolidated balance sheets, financial instruments	(1,534,454)	(2,824,801)
Repurchase agreements, gross amounts not offset in the consolidated balance sheets, cash collateral	(13,162)	0
Repurchase agreements, net amount	6,099	0
Total liabilities
Total liabilities, gross amounts	1,597,682	2,831,171
Total liabilities, gross amounts offset in the consolidated balance sheet	0	0
Total liabilities, net amounts presented in the consolidated balance sheets	1,597,682	2,831,171
Total liabilities, gross amounts not offset in the consolidated balance sheets, financial instruments	(1,552,883)	(2,827,377)
Total liabilities, gross amounts not offset in the consolidated balance sheets, cash collateral	(38,700)	(2,819)
Total liabilities, net amount	6,099	975
Cash pledged to derivative counterparties	27,100	55,400
Fair value of investments pledged against repurchase agreements	$ 1,700,000	$ 3,200,000