Mortgage-Backed Securities and other securities - Purchase discount and premium (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Discount Designated as Credit Reserve and OTTI
Other-than-temporary Impairment Loss, Debt Securities, Portion Recognized in Earnings	$ 0	$ (1,232)
Non-Agency RMBS and Non-Agency CMBS and other securities
Discount Designated as Credit Reserve and OTTI
Balance at beginning of period		(53,523)
Realized credit losses		3,001
Sales		2,694
Other-than-temporary Impairment Loss, Debt Securities, Portion Recognized in Earnings		(966)
Transfers/release of credit reserve		(1,293)
Balance at end of period		(50,087)
Accretable Discount
Balance at beginning of period		(29,465)
Accretion of discount		1,277
Transfers/release of credit reserve		(556)
Balance at end of period		(27,632)
Amortizable Premium
Balance at beginning of period		(14,928)
Amortization of premium		(481)
Sales		(523)
Transfers/release of credit reserve		(737)
Balance at end of period		$ (14,661)