Mortgage-Backed Securities and other securities - OTTI and Interest Income (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Debt Securities, Available-for-sale [Line Items]
Other than temporary impairment									$ 8,574	$ 11,180	$ 22,873
Depreciation, Amortization and Accretion, Net [Abstract]
Net (Premium Amortization/Amortization Basis) Discount Amortization									(7,398)	(5,374)	2,070
Interest income	$ 55,761	$ 55,652	$ 53,818	$ 52,033	$ 58,020	$ 54,461	$ 57,154	$ 39,727	217,264	209,362	124,291
Total mortgage-backed securities and other securities
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									93,960	121,638	111,233
Net (Premium Amortization/Amortization Basis) Discount Amortization									(10,049)	(6,816)	(2,191)
Interest income									83,911	114,822	109,042
Subtotal Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Other than temporary impairment									74	807	5,774
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									12,181	19,507	38,108
Net (Premium Amortization/Amortization Basis) Discount Amortization									(3,053)	(5,092)	(13,058)
Interest income									9,128	14,415	25,050
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Other than temporary impairment									1,331	996	0
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									4,682	7,120	5,602
Net (Premium Amortization/Amortization Basis) Discount Amortization									(2,214)	(1,073)	525
Interest income									2,468	6,047	6,127
Agency CMBS
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									51,286	60,148	40,064
Net (Premium Amortization/Amortization Basis) Discount Amortization									(2,327)	(646)	507
Interest income									48,959	59,502	40,571
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Other than temporary impairment									6,565	8,660	15,117
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									14,178	20,058	19,179
Net (Premium Amortization/Amortization Basis) Discount Amortization									4,017	6,366	8,276
Interest income									18,195	26,424	27,455
Other securities
Debt Securities, Available-for-sale [Line Items]
Other than temporary impairment									604	717	1,982
Depreciation, Amortization and Accretion, Net [Abstract]
Coupon Interest									11,633	14,805	8,280
Net (Premium Amortization/Amortization Basis) Discount Amortization									(6,472)	(6,371)	1,559
Interest income									$ 5,161	$ 8,434	$ 9,839