Mortgage-Backed Securities and other securities - Changes in the components of purchase discount and amortizable premium (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Discount Designated as Credit Reserve and OTTI
Other than temporary impairment	$ (2,228)	$ (1,819)	$ (3,295)	$ (1,232)	$ (2,757)	$ (2,533)	$ (2,974)	$ (2,916)
Non-Agency RMBS and Non-Agency CMBS and Other Securities
Discount Designated as Credit Reserve and OTTI
Balance at beginning of period				(53,523)				(72,915)	$ (53,523)	$ (72,915)	$ (130,484)
Realized credit losses									7,290	5,863	2,391
Purchases									(28)	(7,182)	(19,385)
Sales									26,706	32,301	89,628
Other than temporary impairment									(6,612)	(9,733)	(15,310)
Transfers/release of credit reserve									(22,308)	(1,857)	245
Balance at end of period	(48,475)				(53,523)				(48,475)	(53,523)	(72,915)
Accretable Discount
Balance at beginning of period				(29,465)				(68,438)	(29,465)	(68,438)	(109,822)
Accretion of discount									4,364	7,137	10,715
Purchases									(7,953)	(6,473)	(1,205)
Sales									0	40,338	33,166
Transfers/release of credit reserve									2,889	(2,029)	(1,292)
Balance at end of period	(30,165)				(29,465)				(30,165)	(29,465)	(68,438)
Amortizable Premium
Balance at beginning of period				$ 14,928				$ 20,872	14,928	20,872	44,527
Amortization of premium									(1,215)	(675)	(843)
Purchases									819	435	7,259
Sales									(19,640)	(9,590)	(31,118)
Transfers/release of credit reserve									19,419	3,886	1,047
Balance at end of period	$ 14,311				$ 14,928				$ 14,311	$ 14,928	$ 20,872