CONSOLIDATED STATEMENTS OF CASH FLOWS - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Operating Activities:
Net (loss) income	$ (55,419)	$ (6,488)	$ 54,372
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
Depreciation on rental properties	478	473	465
Realized loss on loans held-for-securitization			(378)
Impairment charge	2,108	2,869	2,399
(Gain) on sales of residential mortgage loans			(378)
(Gain) on sale of residential properties	(31)	(54)
Gain on sale of residential properties	(31)	(54)
Amortization of restricted stock	78	98	82
Recovery on Non-Agency MBS		(1)	(2)
Net settlements received on interest rate swaps, net of amortization	11,459	7,851	(6,714)
Unrealized loss on interest rate swaps, net	98,907	(585)	(8,310)
(Gain) loss on derivatives	84,741	8,071	(7,132)
Changes in assets and liabilities:
Decrease in reverse repurchase agreements	5,000	(20,000)
Decrease (increase) in interest receivable	1,158	(983)	(1,394)
(Increase) decrease in prepaid expenses and other	(8,272)	387	415
(Decrease) increase in accrued interest payable	(3,836)	11,085	7,721
Increase in accrued expenses and payables	2,190	(548)	(787)
Net cash provided by operating activities	64,667	66,337	86,068
Investing Activities:
Proceeds from sales, MBS portfolios	2,950,885	787,260	736,900
Purchases, MBS portfolios	(3,603,083)	(1,146,699)	(2,393,106)
Principal payments, MBS portfolios	909,244	937,075	1,148,931
Purchases, Residential mortgage loans held-for-securitization	(179,455)
Principal payments, Residential mortgage loans held-for-securitization	30,992
Residential properties purchases	(362)	(241)	(346)
Proceeds from sales	95	203
Net cash (used in) provided by investing activities	108,437	577,712	(500,681)
Financing Activities:
Borrowings from repurchase agreements	32,300,305	23,720,955	26,868,172
Repayments on repurchase agreements	(32,454,059)	(24,275,023)	(26,413,492)
Borrowings from warehouse line of credit	156,037
Repayments on warehouse line of credit	(22,210)
Net settlements of TBA Agency MBS Contracts	19,650	(15,956)	(13,259)
Termination of de-designated interest rate swaps	(39,226)		(27,350)
Derivative counterparty margin	367
Proceeds from common stock issued	1,361	1,627	3,790
Preferred Stock dividends paid	(9,189)	(9,169)	(7,555)
Common stock dividends paid	(46,348)	(56,977)	(57,996)
Net cash provided by (used in) financing activities	(93,630)	(634,018)	394,622
Net (decrease) increase in cash, cash equivalents, and restricted cash	79,474	10,031	(19,991)
Cash, cash equivalents, and restricted cash at beginning of period	33,461	23,430	43,421
Cash, cash equivalents, and restricted cash at end of period	112,935	33,461	23,430
Supplemental Disclosure of Cash Flow Information:
Cash paid for interest	94,691	75,318	58,003
Change in payables for residential mortgage loans purchased	(6,115)
Residential mortgage loans
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
Amortization of premium (discount)	(116)	(112)	(150)
Impairment charge		18
Investing Activities:
Proceeds from sales, Residential mortgage loans held-for investment			6,806
Principal payments, Residential mortgage loans held-for-investment	121	114	134
Residential loans
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
Amortization of premium (discount)	298
Agency MBS
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
Amortization of premium (discount)	26,820	28,800	34,080
Realized (gain) loss on sales of available-for-sale MBS	3,983	12,186	2,168
Realized loss on sales of Agency MBS held as trading investments	5,787	11,429
Unrealized (gain) loss on Agency MBS held as trading investments	(17,036)	4,911	(2,793)
Investing Activities:
Proceeds from sales, MBS portfolios	2,950,000	781,000
Non-Agency MBS
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
Amortization of premium (discount)	5,277	6,345	3,454
Impairment charge	2,108	2,869	2,399
Investing Activities:
Proceeds from sales, MBS portfolios		6,000
Series C Preferred Stock
Financing Activities:
Proceeds (amortization) on Series C Preferred Stock issued	(318)	525	42,312
TBA Agency MBS
Adjustments to reconcile net (loss) income to net cash provided by operating activities:
(Gain) loss on derivatives	$ (14,166)	$ 8,656	$ 1,440