CONSOLIDATED STATEMENTS OF CASH FLOWS - USD ($) $ in Thousands	3 Months Ended				6 Months Ended		12 Months Ended
	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2019	Mar. 31, 2019	Jun. 30, 2020	Jun. 30, 2019	Dec. 31, 2019
Operating Activities:
Net income (loss)	$ 36,839	$ (185,821)	$ (47,700)	$ (19,970)	$ (148,981)	$ (67,670)
Adjustments to reconcile net (loss) to net cash provided by operating activities:
Depreciation on rental properties	119		119		240	238
Realized gain (loss) on sale of available-for-sale MBS	(10,095)		(444)		(15,805)	5,703
(Gain) on sale of residential properties	(45)				(123)
Amortization of restricted stock	4		25		8	50
Net settlements received on interest rate swaps, net of amortization	964		4,395		1,853	9,256
Loss on interest rate swaps, net	4,990		57,710		104,599	91,429
(Gain) on derivatives, net of derivative income - TBA Agency MBS	6,168		53,543		94,958	80,832
Changes in assets and liabilities:
Decrease in reverse repurchase agreements					15,000	20,000
Decrease (increase) in interest receivable	3,100		1,932		7,054	1,167
Decrease (increase) in prepaid expenses and other	3,885		1,048		6,091	(3,498)
(Decrease) in accrued interest payable	(5,080)		6,514		(5,402)	(2,191)
Increase in accrued expenses and payables	94		12,097		2,044	14,161
Net cash provided by operating activities	11,925		39,920		51,722	65,975
Investing Activities:
Proceeds from sales, MBS portfolios	539,588		1,335,932		1,709,950	2,239,754
Purchases, MBS portfolios			(933,121)		(76,283)	(1,930,488)
Principal payments, MBS portfolios	207,420		248,845		408,040	434,537
Purchases, Residential mortgage loans held-for-securitization			(102,467)		(4,761)	(120,495)
Principal payments, Residential mortgage loans held-for-securitization	10,105		3,674		20,155	4,476
Residential properties purchases	(44)		(25)		(165)	(220)
Proceeds from sales of residential properties	179				428
Net cash provided by investing activities	757,357		552,868		2,057,505	627,624
Financing Activities:
Borrowings from repurchase agreements	5,835,618		7,324,227		12,130,656	14,531,967
Repayments on repurchase agreements	(6,611,571)		(7,929,018)		(14,091,348)	(15,187,751)
Borrowings from warehouse line of credit			80,006			95,722
Repayments on warehouse line of credit	(11,599)		(3,023)		(29,029)	(3,023)
Net settlements of TBA Agency MBS Contracts	3,373		6,986		13,375	15,008
Termination of interest rate swaps	(14,858)		(10,385)		(62,895)	(10,385)
Derivative counterparty margin	(4,578)		(4,635)		1,689	603
Proceeds from common stock issued	159		402		425	758
Proceeds (amortization) on Series C Preferred Stock issued			(318)			(318)
Preferred Stock dividends paid	(2,297)		(2,297)		(4,595)	(4,594)
Common stock dividends paid	(4,947)		(12,813)		(13,844)	(25,616)
Net cash (used in) financing activities	(810,700)		(550,868)		(2,055,566)	(587,629)
Net increase in cash, cash equivalents, and restricted cash	(41,418)		41,920		53,661	105,970
Cash, cash equivalents, and restricted cash at beginning of period	208,014	112,935	97,511	33,461	112,935	33,461	$ 33,461
Cash, cash equivalents, and restricted cash at end of period	166,596	$ 208,014	139,431	$ 97,511	166,596	139,431	$ 112,935
Supplemental Disclosure of Cash Flow Information:
Cash paid for interest	15,510		17,694		39,972	49,306
Change in payable for MBS purchased			(227,997)
Change in payables for residential mortgage loans purchased			(102,938)		(5,545)	(2,282)
Change in receivable for securities sold	(65,828)
Residential mortgage loans
Adjustments to reconcile net (loss) to net cash provided by operating activities:
Amortization of premium (discount)	(29)		(29)		(59)	(58)
Provision for credit losses on loans	564				564
Investing Activities:
Principal payments, Residential mortgage loans held-for-investment through consolidated securitization trusts	109		30		141	60
Residential loans
Adjustments to reconcile net (loss) to net cash provided by operating activities:
Amortization of premium (discount)	404		16		730	16
Agency MBS
Adjustments to reconcile net (loss) to net cash provided by operating activities:
Amortization of premium (discount)	4,382		7,548		10,859	13,434
Realized gain (loss) on sale of available-for-sale MBS	(10,095)		(444)		(15,805)	5,703
(Gain) loss on sales of MBS held as trading investments			(234)		(3,981)	7,128
Unrealized loss (gain) on Agency MBS held as trading investments			(989)		1,141	(15,895)
Investing Activities:
Proceeds from sales, MBS portfolios	400,000		1,340,000		1,400,000	2,240,000
Non-Agency MBS
Adjustments to reconcile net (loss) to net cash provided by operating activities:
Amortization of premium (discount)			1,473		1,253	2,695
(Gain) loss on sales of MBS held as trading investments	(25,687)				89,686
Impairment charge			606			606
Investing Activities:
Proceeds from sales, MBS portfolios			0
TBA Agency MBS
Adjustments to reconcile net (loss) to net cash provided by operating activities:
(Gain) on derivatives, net of derivative income - TBA Agency MBS	$ (2,484)		$ (4,167)		$ (15,049)	$ (10,596)