Consolidated Statements of Cash Flows $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($)	Dec. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)
Statement of Cash Flows [Abstract]
Net income (loss)	$ 85,097	$ (25,015)	$ (9,484)
Adjustments to reconcile net income (loss) to net cash provided by operating activities:
Premium amortization and (discount accretion) on investments, net	(2,070)	(548)	5,086
Interest income earned added to principal of securities	(46)	(413)	(207)
Amortization of deferred financing costs	192	191	486
Amortization of discount on convertible senior unsecured notes	137	0	0
Restricted stock amortization	981	1,699	2,301
Premium amortization for MAC interest rate swaps	0	(658)	(1,250)
Interest payments and basis recovered on MAC interest rate swaps	525	658	1,595
Premium on purchase of Residential Whole-Loans	(1,239)	(574)	(3,329)
Premium on purchase of Residential Bridge Loans	(753)	0	0
Unrealized (gain) loss, net	(28,396)	17,107	34,011
Unrealized loss on derivative instruments, net	(148,308)	5,135	73,599
Other than temporary impairment	22,873	32,286	19,791
Realized (gain) loss on sale of investments, net	(20,598)	21,991	(8,279)
(Gain) loss on derivatives, net	156,076	(41,516)	13,920
Loss (gain) on foreign currency transactions, net	1	758	(2,130)
Changes in operating assets and liabilities:
Decrease (increase) in accrued interest receivable	5,209	3,809	4,688
Decrease (increase) in other assets	(1,763)	(16)	(56)
Increase (decrease) in accrued interest payable	(7,719)	(4,390)	2,858
Increase (decrease) in accounts payable and accrued expenses	(274)	1,477	(16)
Increase (decrease) in payable to related party	(543)	(435)	314
Net cash provided by operating activities	59,382	11,546	133,898
Cash flows from investing activities:
Purchase of securities	(2,881,276)	(2,172,623)	(1,354,274)
Proceeds from sale of securities	1,593,914	2,273,922	2,547,364
Principal repayments and basis recovered on securities	240,785	344,937	397,073
Purchase of Residential Whole-Loans	(87,640)	(28,825)	(225,870)
Principal repayments on Residential Whole-Loans	42,176	52,473	20,355
Purchase of Commercial Whole-Loans	0	0	(8,750)
Principal repayments on Commercial Whole-Loans	0	0	8,750
Purchase of securitized commercial loan	0	0	(14,000)
Principal repayments on securitized commercial loan	154	0	0
Purchase of Residential Bridge Loans	(142,913)	0	0
Purchase of Residential Bridge Loans	31,794	0	0
Payment of premium for option derivatives	(16,355)	(17,951)	(16,064)
Premium received from option derivatives	14,553	22,707	15,354
Premium received from credit default swap	74	0	0
Net settlements of TBAs	4,035	6,140	3,710
Proceeds from (Payments on) termination of futures, net	(9,130)	23,609	(527)
Proceeds from sale of interest rate swaptions	0	2,075	32,186
Premium for MAC interest rate swaps, net	0	465	(3,858)
Payments on termination of MAC interest rate swaps	0	0	(186)
Due from counterparties	8,449	(8,449)	0
Proceeds from termination of foreign currency swaps	0	6,771	0
Payments on total return swaps	(552)	38	0
Interest payments and basis recovered on MAC interest rate swaps	(525)	(658)	(1,193)
Premium for interest rate swaptions, net	(115)	0	(40,215)
Net cash provided by (used in) investing activities	(1,202,572)	504,631	1,359,855
Cash flows from financing activities:
Payment of offering costs	(94)	0	0
Repurchase of common stock	(1,094)	0	0
Proceeds from repurchase agreement borrowings	18,438,166	15,977,505	17,544,060
Proceeds from convertible note offering	115,000	0	0
Repayments of repurchase agreement borrowings	(17,342,124)	(16,407,929)	(18,863,913)
Proceeds from (Repayment of) cash overdraft	0	(300)	300
Repayments of securitized debt	(68)	0	0
Proceeds from forward contracts	8,246	112,574	242,279
Repayments of forward contracts	(8,214)	(112,640)	(243,180)
Payments on termination of MAC interest rate swaps containing an other-than-insignificant financing element	0	0	(18,655)
Interest payments and basis recovered on MAC interest rate swaps containing an other-than-insignificant financing element	0	0	(402)
Payments made for deferred financing costs	(3,837)	(58)	(620)
Due from counterparties, net	(9,707)	14,427	(64,806)
Due to counterparties, net	750	(9,210)	(2,230)
Dividends paid on common stock	(51,981)	(69,168)	(109,272)
Net cash provided by (used in) financing activities	1,145,043	(494,799)	(1,516,439)
Effect of exchange rate changes on cash and cash equivalents	(1)	83	175
Net increase (decrease) in cash and cash equivalents	1,852	21,461	(22,511)
Cash and cash equivalents beginning of period	46,172	24,711	47,222
Cash and cash equivalents end of period	48,024	46,172	24,711
Supplemental disclosure of operating cash flow information:
Interest paid	43,032	32,041	27,515
Income taxes paid	4,966	2,093	0
Supplemental disclosure of non-cash financing/investing activities:
Repurchase of common stock, not settled	(137)	0	0
Principal payments of securities, not settled	0	0	572
Securities purchased, not settled	(17,217)	(309,086)	(66,146)
Assets of CSMC Trust	0	0	11,000
Liabilities of CSMC Trust	0	0	(11,000)
Net unsettled TBAs	0	(14)	0
Derivative collateral offset against derivatives	(157,913)	0	0
Dividends and distributions declared, not paid	12,960	12,995	24,313
Principal payments of Residential Whole-Loans, not settled	2,286	1,241	0
Principal payments of Residential Bridge Loans, not settled	$ 5,381	$ 0	$ 0