Mortgage-Backed Securities and other securities - Company's investment portfolio (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Total mortgage-backed securities and other securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 2,154,864	$ 1,836,035
Unamortized Premium (Discount), net	16,133	(8,717)
Discount Designated as Credit Reserve and OTTI	(48,475)	(53,523)
Amortized Cost	2,149,607	1,813,403
Unrealized Gain	90,911	24,521
Unrealized Loss	(11,933)	(33,043)
Estimated Fair Value	$ 2,237,249	$ 1,816,741
Net Weighted Average Coupon (as a percent)	3.10%	2.90%
Weighted average expected remaining term to the expected maturity of investment portfolio	7 years 10 months 24 days	8 years 6 months
Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 1,347,929	$ 1,493,675
Unamortized Premium (Discount), net	26,514	5,820
Amortized Cost	1,374,443	1,499,495
Unrealized Gain	66,832	12,083
Unrealized Loss	(5,798)	(29,594)
Estimated Fair Value	$ 1,435,477	$ 1,481,984
Net Weighted Average Coupon (as a percent)	3.40%	3.30%
Agency CMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 3,092	$ 4,158
Net Weighted Average Coupon (as a percent)	0.40%	0.40%
Notional balance	$ 160,200	$ 172,200
Subtotal Commercial Mortgage Backed Securities Issued By US Government Sponsored Enterprises
Debt Securities, Available-for-sale [Line Items]
Principal Balance	1,347,929	1,493,675
Unamortized Premium (Discount), net	26,514	5,820
Amortized Cost	1,374,443	1,499,495
Unrealized Gain	66,832	12,083
Unrealized Loss	(5,798)	(29,594)
Estimated Fair Value	$ 1,438,569	$ 1,486,142
Net Weighted Average Coupon (as a percent)	3.10%	3.00%
Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 327,814
Unamortized Premium (Discount), net	5,473
Amortized Cost	333,287
Unrealized Gain	7,484
Unrealized Loss	0
Estimated Fair Value	$ 340,771
Net Weighted Average Coupon (as a percent)	3.50%
Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 8,661	$ 11,480
Unrealized Gain	1,820	1,062
Unrealized Loss	(138)	(407)
Estimated Fair Value	$ 10,343	$ 12,135
Net Weighted Average Coupon (as a percent)	2.80%	2.20%
Notional balance	$ 121,700	$ 158,800
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 5,572	$ 7,702
Net Weighted Average Coupon (as a percent)	3.00%	2.90%
Subtotal Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 327,814
Unamortized Premium (Discount), net	5,473
Amortized Cost	341,948	$ 11,480
Unrealized Gain	9,304	1,062
Unrealized Loss	(138)	(407)
Estimated Fair Value	$ 356,686	$ 19,837
Net Weighted Average Coupon (as a percent)	3.30%	2.50%
Subtotal Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 1,675,743	$ 1,493,675
Unamortized Premium (Discount), net	31,987	5,820
Amortized Cost	1,716,391	1,510,975
Unrealized Gain	76,136	13,145
Unrealized Loss	(5,936)	(30,001)
Estimated Fair Value	$ 1,795,255	$ 1,505,979
Net Weighted Average Coupon (as a percent)	3.10%	2.90%
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 52,767	$ 54,887
Unamortized Premium (Discount), net	4,492	6,909
Discount Designated as Credit Reserve and OTTI	(20,256)	(23,731)
Amortized Cost	37,003	38,065
Unrealized Gain	1,388	961
Unrealized Loss	(260)	0
Estimated Fair Value	$ 38,131	$ 39,026
Net Weighted Average Coupon (as a percent)	4.80%	4.80%
Non-Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 7,705	$ 11,154
Unrealized Gain	636	382
Unrealized Loss	(658)	(7)
Estimated Fair Value	$ 7,683	$ 11,529
Net Weighted Average Coupon (as a percent)	0.60%	0.60%
Notional balance	$ 442,400	$ 519,900
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	5,572	7,702
Notional balance	64,800	89,800
Subtotal Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	52,767	54,887
Unamortized Premium (Discount), net	4,492	6,909
Discount Designated as Credit Reserve and OTTI	(20,256)	(23,731)
Amortized Cost	44,708	49,219
Unrealized Gain	2,024	1,343
Unrealized Loss	(918)	(7)
Estimated Fair Value	$ 45,814	$ 50,555
Net Weighted Average Coupon (as a percent)	1.00%	1.00%
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 354,458	$ 240,431
Unamortized Premium (Discount), net	(17,909)	(20,317)
Discount Designated as Credit Reserve and OTTI	(22,016)	(22,189)
Amortized Cost	314,533	197,925
Unrealized Gain	6,359	5,021
Unrealized Loss	(4,873)	(2,645)
Estimated Fair Value	$ 316,019	$ 200,301
Net Weighted Average Coupon (as a percent)	5.10%	5.90%
Total Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 407,225	$ 295,318
Unamortized Premium (Discount), net	(13,417)	(13,408)
Discount Designated as Credit Reserve and OTTI	(42,272)	(45,920)
Amortized Cost	359,241	247,144
Unrealized Gain	8,383	6,364
Unrealized Loss	(5,791)	(2,652)
Estimated Fair Value	$ 361,833	$ 250,856
Net Weighted Average Coupon (as a percent)	2.70%	2.40%
Other securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 71,896	$ 47,042
Unamortized Premium (Discount), net	(2,437)	(1,129)
Discount Designated as Credit Reserve and OTTI	(6,203)	(7,603)
Amortized Cost	73,975	55,284
Unrealized Gain	6,392	5,012
Unrealized Loss	(206)	(390)
Estimated Fair Value	$ 80,161	$ 59,906
Net Weighted Average Coupon (as a percent)	6.70%	9.00%
Residual interests in asset-backed securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance		$ 0
Amortized Cost	$ 10,700	$ 17,000