Mortgage-Backed Securities and other securities - Changes in the components of purchase discount and amortizable premium (Details) - Non-Agency RMBS and Non-Agency CMBS and Other Securities $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($)
Discount Designated as Credit Reserve and OTTI
Balance at beginning of period	$ (53,523)
Realized credit losses	7,290
Purchases	(28)
Sales	(26,706)
Net impairment losses recognized in earnings	(6,612)
Transfers/release of credit reserve	(22,308)
Balance at end of period	48,475
Accretable Discount
Balance at beginning of period	(29,465)
Accretion of discount	(4,364)
Purchases	(7,953)
Sales	0
Transfers/release of credit reserve	2,889
Balance at end of period	30,165
Amortizable Premium
Balance at beginning of period	14,928
Amortization of premium	(1,215)
Purchases	819
Sales	(19,640)
Transfers/release of credit reserve	19,419
Balance at end of period	$ (14,311)