Mortgage-Backed Securities and other securities - Company's Investment Portfolio (Details) - USD ($)	6 Months Ended	12 Months Ended
	Jun. 30, 2022	Dec. 31, 2021
Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 53,000	$ 59,000
Unrealized Gain	4,000	55,000
Unrealized Loss	0	0
Estimated Fair Value	$ 57,000	$ 114,000
Net Weighted Average Coupon	0.50%	1.30%
Notional balance	$ 2,600,000	$ 2,900,000
Agency RMBS Interest-Only Strips, accounted for as derivatives
Debt Securities, Available-for-sale [Line Items]
Estimated Fair Value	$ 728,000	$ 1,058,000
Net Weighted Average Coupon	1.50%	1.30%
Notional balance	$ 14,600,000	$ 16,800,000
Total Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	0	0
Unamortized Premium (Discount), net	0	0
Amortized Cost	53,000	59,000
Unrealized Gain	4,000	55,000
Unrealized Loss	0	0
Estimated Fair Value	$ 785,000	$ 1,172,000
Net Weighted Average Coupon	1.40%	1.30%
Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 45,844,000	$ 36,147,000
Unamortized Premium (Discount), net	(15,074,000)	(13,936,000)
Amortized Cost	30,770,000	22,211,000
Unrealized Gain	2,827,000	3,476,000
Unrealized Loss	(2,580,000)	(35,000)
Estimated Fair Value	$ 31,017,000	$ 25,652,000
Net Weighted Average Coupon	4.20%	4.30%
Non-Agency RMBS Interest-Only Strips
Debt Securities, Available-for-sale [Line Items]
Amortized Cost	$ 5,402,000	$ 5,608,000
Unrealized Gain	0	0
Unrealized Loss	(4,221,000)	(3,491,000)
Estimated Fair Value	$ 1,181,000	$ 2,117,000
Net Weighted Average Coupon	0.20%	0.30%
Notional balance	$ 153,500,000	$ 181,000,000
Subtotal Non-Agency RMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	45,844,000	36,147,000
Unamortized Premium (Discount), net	(15,074,000)	(13,936,000)
Amortized Cost	36,172,000	27,819,000
Unrealized Gain	2,827,000	3,476,000
Unrealized Loss	(6,801,000)	(3,526,000)
Estimated Fair Value	$ 32,198,000	$ 27,769,000
Net Weighted Average Coupon	1.10%	1.00%
Non-Agency CMBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 114,148,000	$ 179,619,000
Unamortized Premium (Discount), net	(3,378,000)	(13,088,000)
Amortized Cost	110,770,000	166,531,000
Unrealized Gain	982,000	1,543,000
Unrealized Loss	(18,656,000)	(62,716,000)
Estimated Fair Value	$ 93,096,000	$ 105,358,000
Net Weighted Average Coupon	6.70%	5.40%
Total Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 159,992,000	$ 215,766,000
Unamortized Premium (Discount), net	(18,452,000)	(27,024,000)
Amortized Cost	146,942,000	194,350,000
Unrealized Gain	3,809,000	5,019,000
Unrealized Loss	(25,457,000)	(66,242,000)
Estimated Fair Value	$ 125,294,000	$ 133,127,000
Net Weighted Average Coupon	3.10%	3.00%
Other securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 46,517,000	$ 51,159,000
Unamortized Premium (Discount), net	(8,586,000)	(8,229,000)
Amortized Cost	42,772,000	47,652,000
Unrealized Gain	2,025,000	4,209,000
Unrealized Loss	(4,263,000)	(213,000)
Estimated Fair Value	$ 40,534,000	$ 51,648,000
Net Weighted Average Coupon	6.10%	5.60%
Total
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 206,509,000	$ 266,925,000
Unamortized Premium (Discount), net	(27,038,000)	(35,253,000)
Amortized Cost	189,767,000	242,061,000
Unrealized Gain	5,838,000	9,283,000
Unrealized Loss	(29,720,000)	(66,455,000)
Estimated Fair Value	$ 166,613,000	$ 185,947,000
Net Weighted Average Coupon	3.40%	3.20%
Weighted average expected remaining term to the expected maturity of investment portfolio	7 years	5 years 10 months 24 days
Residual interests in asset-backed securities
Debt Securities, Available-for-sale [Line Items]
Principal Balance	$ 0	$ 0
Amortized Cost	$ 4,800,000	$ 4,700,000