Consolidated Statements of Cash Flows $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)	Dec. 31, 2017 USD ($)
Statement of Cash Flows [Abstract]
Net income (loss)	$ 70,699	$ 26,409	$ 85,097
Adjustments to reconcile net income (loss) to net cash provided by operating activities:
Premium amortization and (discount accretion), net	7,398	5,374	(2,070)
Interest income earned added to principal of securities	0	0	(46)
Amortization of deferred financing costs	1,488	767	192
Amortization of Debt Discount (Premium)	718	550	137
Restricted stock amortization	564	265	981
Interest payments and basis recovered on MAC interest rate swaps	5,772	2,465	525
Premium on purchase of Residential Whole Loans	(15,304)	(15,850)	(1,239)
Premium on purchase of Residential Bridge Loans	0	(3,889)	(753)
Premium on purchase of securitized commercial loans	(3,769)	(3,019)	0
Unrealized (gain) loss, net	(107,529)	24,671	(28,396)
Unrealized (gain) loss on derivative instruments, net	(9,390)	10,381	(148,308)
Other than temporary impairment	8,574	11,180	22,873
Realized loss on sale of real estate owned (REO), net	90	0	0
Realized (gain) loss on sale of investments, net	(28,368)	63,257	(20,598)
Other Settlements / Expirations	9,631	(10,316)	156,076
Loss on foreign currency transactions, net	0	0	1
Changes in operating assets and liabilities:
Decrease (increase) in interest receivable	2,546	(8,356)	5,209
Decrease (increase) in other assets	1,157	(184)	(1,763)
Increase (decrease) in interest payable	6,469	210	(7,719)
(Decrease) increase in accounts payable and accrued expenses	(519)	563	(274)
(Decrease) increase in payable to related party	(2,467)	2,574	(543)
Net cash (used in) provided by operating activities	(52,240)	107,052	59,382
Cash flows from investing activities:
Purchase of securities	(1,581,485)	(1,128,399)	(2,881,276)
Proceeds from sale of securities	1,136,617	2,430,054	1,593,914
Principal repayments and basis recovered on securities	126,091	123,837	240,785
Proceeds from sale of REO	1,033	0	0
Purchase of Residential Whole Loans	(563,821)	(873,911)	(87,640)
Principal repayments on Residential Whole Loans	254,125	76,942	42,176
Purchase of commercial loans	(350,232)	(235,857)	0
Principal repayments on commercial loans	197,245	20,638	0
Purchase of securitized commercial loans	(1,109,461)	(1,350,000)	0
Principal repayments on securitized commercial loans	1,214,688	361,781	154
Purchase of Residential Bridge Loans	0	(418,953)	(142,913)
Principal repayments on Residential Bridge Loans	211,316	274,964	31,794
Payment of premium for option derivatives	(780)	(1,757)	(16,355)
Premium received from option derivatives	2,158	1,236	14,553
Premium for credit default swaps, net	3,885	(297)	74
Net settlements of TBAs	1,934	(800)	4,035
Proceeds from (Payments on) termination of futures, net	(12,862)	6,112	(9,130)
Due from counterparties, net	(2,850)	0	8,449
Payments on total return swaps	0	0	(552)
Interest payments and basis recovered on MAC interest rate swaps	(5,772)	(2,465)	(525)
Premium for interest rate swaptions, net	(332)	0	(115)
Net cash used in investing activities	(478,503)	(716,875)	(1,202,572)
Cash flows from financing activities:
Net proceeds from issuance of common stock	52,714	64,880	0
Payment of offering costs	(580)	(195)	(94)
Repurchase of common stock	0	(1,733)	(1,094)
Proceeds from sale of treasury stock	0	3,177	0
Proceeds from repurchase agreement borrowings	21,381,571	21,472,963	18,438,166
Proceeds from convertible note offering	90,625	0	115,000
Repayments of repurchase agreement borrowings	(21,375,531)	(21,905,812)	(17,342,124)
Proceeds from securitized debt	1,828,361	1,285,219	0
Repayments of securitized debt	(1,292,141)	(344,612)	(68)
Proceeds from forward contracts	0	0	8,246
Repayments of forward contracts	0	0	(8,214)
Payments made for deferred financing costs	(8,522)	0	(3,837)
Due from counterparties, net	(56,474)	47,307	(9,707)
Due to counterparties, net	(17,072)	16,291	750
Increase (decrease) in other liabilities, net	(2,860)	55,808	0
Dividends paid on common stock	(62,864)	(53,699)	(51,981)
Net cash provided by financing activities	537,227	639,594	1,145,043
Effect of exchange rate changes on cash and cash equivalents	0	0	(1)
Net increase in cash and cash equivalents	6,484	29,771	1,852
Cash, cash equivalents and restricted cash, beginning of period	77,795	48,024	46,172
Cash, cash equivalents and restricted cash, end of period	84,279	77,795	48,024
Supplemental disclosure of operating cash flow information:
Interest paid	144,987	138,524	43,032
Income taxes paid	549	1,635	4,966
Supplemental disclosure of non-cash financing/investing activities:
Underwriting and offering costs payable	27	177	0
Repurchase of common stock, not settled	0	0	(137)
Securities purchased, not settled	0	0	(17,217)
Derivative collateral offset against derivatives	0	0	(157,913)
Dividends and distributions declared, not paid	16,592	14,916	12,960
Principal payments of Residential Whole Loans, not settled	17,254	9,230	2,286
Principal payments of Residential Bridge Loans, not settled	1,949	33,717	5,381
Other assets - Transfer of Bridge Loans to REO	5,029	143	0
Proceeds from sale of REO, not settled	728	0	0
Cash, Cash Equivalents, Restricted Cash and Restricted Cash Equivalents [Abstract]
Cash and cash equivalents	31,331	21,987	48,024
Restricted cash	$ 52,948	$ 55,808	$ 0