Mortgage-Backed Securities and other securities - Type of security (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 413,394	$ 1,435,477
Agency CMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	68,875	973,189
Agency CMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	344,519	462,288
Agency CMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips, accounted for as derivatives
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	2,792	3,092
Agency CMBS Interest-Only Strips, accounted for as derivatives | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips, accounted for as derivatives | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips, accounted for as derivatives | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips, accounted for as derivatives | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	2,792	3,092
Subtotal Agency
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value		340,771
Subtotal Agency | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value		0
Subtotal Agency | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value		0
Subtotal Agency | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value		340,771
Subtotal Agency | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value		0
Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	9,952	10,343
Agency RMBS Interest-Only Strips | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,965	2,413
Agency RMBS Interest-Only Strips | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,924	1,966
Agency RMBS Interest-Only Strips | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	6,063	5,964
Agency RMBS Interest-Only Strips | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	4,490	5,572
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	545	669
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	3,192	3,893
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	753	1,010
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	430,628	1,795,255
Agency MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	71,385	976,271
Agency MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	349,635	468,147
Agency MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	6,816	347,745
Agency MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	2,792	3,092
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	250,309	316,019
Non-Agency CMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	69,336	89,782
Non-Agency CMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	116,300	125,282
Non-Agency CMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	63,839	92,610
Non-Agency CMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	834	8,345
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	21,123	38,131
Non-Agency RMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	7,732	8,966
Non-Agency RMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	13,391	29,165
Non-Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	5,174	7,683
Non-Agency RMBS Interest-Only Strips | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS Interest-Only Strips | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS Interest-Only Strips | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	744	1,716
Non-Agency RMBS Interest-Only Strips | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	4,430	5,967
Subtotal Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	276,606	361,833
Subtotal Non-Agency MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	69,336	89,782
Subtotal Non-Agency MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	116,300	125,282
Subtotal Non-Agency MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	72,315	103,292
Subtotal Non-Agency MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	18,655	43,477
Other securities
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	47,411	80,161
Other securities | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	13,168	25,824
Other securities | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	10,937	31,823
Other securities | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	13,126	2,768
Other securities | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	10,180	19,746
Total
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	754,645	2,237,249
Total | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	153,889	1,091,877
Total | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	476,872	625,252
Total | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	92,257	453,805
Total | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 31,627	$ 66,315