Mortgage-Backed Securities and other securities - Company's investment portfolio (Details) - USD ($)	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020
Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 59,000	$ 89,000
Unrealized Gain	55,000	54,000
Unrealized Loss	0	0
Estimated Fair Value	$ 114,000	$ 143,000
Net Weighted Average Coupon (as a percent)	1.30%	2.10%
Notional balance	$ 2,900,000	$ 3,700,000
Agency RMBS Interest-Only Strips, accounted for as derivatives
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 1,058,000	$ 1,565,000
Net Weighted Average Coupon (as a percent)	1.30%	2.60%
Subtotal Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 0	$ 0
Unamortized Premium (Discount), net	0	0
Amortized Cost	59,000	89,000
Unrealized Gain	55,000	54,000
Unrealized Loss	0	0
Estimated Fair Value	$ 1,172,000	$ 1,708,000
Net Weighted Average Coupon (as a percent)	1.30%	2.50%
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 36,147,000	$ 38,112,000
Unamortized Premium (Discount), net	(13,936,000)	(14,649,000)
Amortized Cost	22,211,000	23,463,000
Unrealized Gain	3,476,000	451,000
Unrealized Loss	(35,000)	(2,498,000)
Estimated Fair Value	$ 25,652,000	$ 21,416,000
Net Weighted Average Coupon (as a percent)	4.30%	1.60%
Non-Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 5,608,000	$ 6,271,000
Unrealized Gain	0	0
Unrealized Loss	(3,491,000)	(2,306,000)
Estimated Fair Value	$ 2,117,000	$ 3,965,000
Net Weighted Average Coupon (as a percent)	0.30%	0.40%
Notional balance	$ 181,000,000	$ 306,000,000
Subtotal Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	36,147,000	38,112,000
Unamortized Premium (Discount), net	(13,936,000)	(14,649,000)
Amortized Cost	27,819,000	29,734,000
Unrealized Gain	3,476,000	451,000
Unrealized Loss	(3,526,000)	(4,804,000)
Estimated Fair Value	$ 27,769,000	$ 25,381,000
Net Weighted Average Coupon (as a percent)	1.00%	0.60%
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 179,619,000	$ 235,497,000
Unamortized Premium (Discount), net	(13,088,000)	(25,258,000)
Amortized Cost	166,531,000	210,239,000
Unrealized Gain	1,543,000	2,850,000
Unrealized Loss	(62,716,000)	(49,008,000)
Estimated Fair Value	$ 105,358,000	$ 164,081,000
Net Weighted Average Coupon (as a percent)	5.40%	5.00%
Total Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 215,766,000	$ 273,609,000
Unamortized Premium (Discount), net	(27,024,000)	(39,907,000)
Amortized Cost	194,350,000	239,973,000
Unrealized Gain	5,019,000	3,301,000
Unrealized Loss	(66,242,000)	(53,812,000)
Estimated Fair Value	$ 133,127,000	$ 189,462,000
Net Weighted Average Coupon (as a percent)	3.00%	2.40%
Other securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 51,159,000	$ 51,537,000
Unamortized Premium (Discount), net	(8,229,000)	(8,239,000)
Amortized Cost	47,652,000	49,420,000
Unrealized Gain	4,209,000	1,152,000
Unrealized Loss	(213,000)	(1,818,000)
Estimated Fair Value	$ 51,648,000	$ 48,754,000
Net Weighted Average Coupon (as a percent)	5.60%	4.40%
Total mortgage-backed securities and other securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 266,925,000	$ 325,146,000
Unamortized Premium (Discount), net	(35,253,000)	(48,146,000)
Amortized Cost	242,061,000	289,482,000
Unrealized Gain	9,283,000	4,507,000
Unrealized Loss	(66,455,000)	(55,630,000)
Estimated Fair Value	$ 185,947,000	$ 239,924,000
Net Weighted Average Coupon (as a percent)	3.20%	2.50%
Weighted average expected remaining term to the expected maturity of investment portfolio	5 years 10 months 24 days	5 years 6 months
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 1,058,000	$ 1,565,000
Notional balance	16,800,000	21,600,000
Residual interests in asset-backed securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	0	0
Amortized Cost	$ 4,700,000	$ 6,100,000