- Fair value and contractual maturities of the Company's investment securities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Total mortgage-backed securities and other securities
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 2,237,249	$ 1,816,741
Total mortgage-backed securities and other securities | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,091,877	1,142,938
Total mortgage-backed securities and other securities | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	625,252	466,292
Total mortgage-backed securities and other securities | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	453,805	94,487
Total mortgage-backed securities and other securities | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	66,315	113,024
Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,435,477	1,481,984
Agency CMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	973,189	1,101,820
Agency CMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	462,288	380,164
Agency CMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	3,092	4,158
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	3,092	4,158
Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	340,771
Agency RMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0
Agency RMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0
Agency RMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	340,771
Agency RMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0
Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	10,343	12,135
Agency RMBS Interest-Only Strips | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	2,413	3,577
Agency RMBS Interest-Only Strips | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,966	2,402
Agency RMBS Interest-Only Strips | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	5,964	6,156
Agency RMBS Interest-Only Strips | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	5,572	7,702
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	669	1,089
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	3,893	4,053
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,010	2,560
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Subtotal Agency MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,795,255	1,505,979
Subtotal Agency MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	976,271	1,106,486
Subtotal Agency MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	468,147	386,619
Subtotal Agency MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	347,745	8,716
Subtotal Agency MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	3,092	4,158
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	38,131	39,026
Non-Agency RMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	8,966	8,540
Non-Agency RMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	29,165	30,486
Non-Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	7,683	11,529
Non-Agency RMBS Interest-Only Strips | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS Interest-Only Strips | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	0	0
Non-Agency RMBS Interest-Only Strips | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	1,716	4,310
Non-Agency RMBS Interest-Only Strips | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	5,967	7,219
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	316,019	200,301
Non-Agency CMBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	89,782	28,754
Non-Agency CMBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	125,282	53,653
Non-Agency CMBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	92,610	72,921
Non-Agency CMBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	8,345	44,973
Subtotal Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	361,833	250,856
Subtotal Non-Agency MBS | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	89,782	28,754
Subtotal Non-Agency MBS | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	125,282	53,653
Subtotal Non-Agency MBS | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	103,292	85,771
Subtotal Non-Agency MBS | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	43,477	82,678
Other securities
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	80,161	59,906
Other securities | Less than or equal to 10 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	25,824	7,698
Other securities | More than 10 years and less than or equal to 20 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	31,823	26,020
Other securities | More than 20 years and less than or equal to 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	2,768	0
Other securities | More than 30 years
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 19,746	$ 26,188