Consolidated Statements of Cash Flows - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Cash flows from operating activities:
Net income (loss)	$ (49,865)	$ (328,346)	$ 70,699
Adjustments to reconcile net income (loss) to net cash provided by (used in) operating activities:
Premium amortization and (discount accretion), net	1,540	5,141	7,398
Interest income earned added to principal of investments	(485)	(829)	0
Amortization of deferred financing costs	5,445	3,577	1,488
Amortization of discount on convertible senior unsecured notes	944	1,097	718
Restricted stock amortization	618	699	564
Interest payments and basis recovered on MAC interest rate swaps	0	202	5,772
Premium on purchase of Residential Whole Loans	(17,058)	(3,858)	(15,304)
Premium on purchase of securitized commercial loans	0	0	(3,769)
Financing fee	8,540	12,000	0
Unrealized (gain) loss, net	46,391	221,387	(107,529)
Unrealized (gain) loss on derivative instruments, net	208	3,953	(9,390)
Other than temporary impairment	0	0	8,574
Realized (gain) loss on exchange and extinguishment of convertible senior notes, net	1,509	(3,644)	0
Realized (gain) loss on sale of real estate owned ("REO"), net	(54)	789	90
Realized (gain) loss on investments, net	9,472	(81,416)	(28,368)
Loss on derivatives, net	0	13,134	9,631
Changes in operating assets and liabilities:
Interest receivable	1,399	5,845	2,546
Invested related receivable	2,546	(2,544)	0
Other assets	339	(905)	1,157
Interest payable	(1,734)	(2,995)	6,469
Accounts payable and accrued expenses	2,095	(499)	(519)
Payable to affiliate	(1,246)	1,023	(2,467)
Other liabilities	(8,539)	8,541	0
Net cash provided by (used in) operating activities	2,065	(147,648)	(52,240)
Cash flows from investing activities:
Purchase of securities	0	(320,996)	(1,581,485)
Proceeds from sale of securities	27,488	2,234,048	1,136,617
Principal repayments and basis recovered on securities	16,704	35,352	126,091
Proceeds from sale of REO	738	2,620	1,033
Purchase of Residential Whole Loans	(410,790)	(109,480)	(563,821)
Proceeds from sale of Residential Whole Loans	0	144,258	0
Principal repayments on Residential Whole Loans	411,605	288,568	254,125
Purchase of commercial loans	0	0	(350,232)
Principal repayments on commercial loans	103,284	44,819	197,245
Purchase of securitized commercial loans	0	0	(1,109,461)
Principal repayments on securitized commercial loans	354,203	349,608	1,214,688
Principal repayments on Residential Bridge Loans	9,374	22,075	211,316
Payment of premium for option derivatives	0	0	(780)
Premium received from option derivatives	0	0	2,158
Premium for credit default swaps, net	0	(14,028)	3,885
Net settlements of TBAs	0	(2,430)	1,934
Proceeds from (Payments on) termination of futures, net	0	0	(12,862)
Due from counterparties, net	50	2,340	(2,850)
Interest payments and basis recovered on MAC interest rate swaps	0	(202)	(5,772)
Premium for interest rate swaptions, net	0	80	(332)
Net cash provided by (used in) investing activities	512,656	2,676,632	(478,503)
Cash flows from financing activities:
Net proceeds from issuance of common stock	0	22,357	52,714
Payment of offering costs	(92)	(374)	(580)
Repurchase of common stock	(1,087)	(578)	0
Proceeds from repurchase agreement borrowings	4,592,689	10,020,593	21,381,571
Proceeds from convertible note offering	86,250	0	90,625
Payments on extinguishment of convertible senior notes	(136,754)	(21,975)	0
Proceeds from offering to non-controlling interest, net of offering costs	0	2	0
Repayments of repurchase agreement borrowings	(4,334,346)	(12,486,624)	(21,375,531)
Proceeds from securitized debt	0	460,787	1,828,361
Repayments of securitized debt	(683,472)	(579,705)	(1,292,141)
Financing fee	(8,540)	(12,000)	0
Payments made for deferred financing costs	(3,573)	(5,437)	(8,522)
Due from counterparties, net	(2,288)	94,280	(56,474)
Due to counterparties, net	(321)	(388)	(17,072)
Increase (decrease) in other liabilities, net	(75,873)	23,185	(2,860)
Dividends paid on common stock	(14,598)	(19,633)	(62,864)
Dividends paid to non-controlling interest	(8)	(8)	0
Net cash (used in) provided by financing activities	(582,013)	(2,505,518)	537,227
Net (decrease) increase in cash and cash equivalents	(67,292)	23,466	6,484
Cash, cash equivalents and restricted cash, beginning of period	107,745	84,279	77,795
Cash, cash equivalents and restricted cash, end of period	40,453	107,745	84,279
Supplemental disclosure of operating cash flow information:
Interest paid	110,707	119,957	144,987
Income taxes paid	192	810	549
Supplemental disclosure of non-cash financing/investing activities:
Underwriting and offering costs payable	2	0	27
Principal payments of securities, not settled	0	44	0
Assets of deconsolidated VIE	0	(150,804)	0
Liabilities of deconsolidated VIE	0	143,952	0
Mortgage-backed securities recorded upon deconsolidation	0	6,852	0
Mortgage-backed securities derecognized upon VIE consolidation	0	(13,737)	0
Dividends and distributions declared, not paid	3,623	3,649	16,592
Principal payments of Residential Whole Loans, not settled	21,970	26,885	17,254
Principal payments of Residential Bridge Loans, not settled	163	1,102	1,949
Other assets - Transfer of Bridge Loans to REO	752	419	5,029
Other assets - Transfer of Commercial Loans to REO	30,345	0	0
Other assets - Transfer of REO from non-controlling interest	12,138	0	0
Proceeds from sale of REO, not settled	0	0	728
Financing fee payable	0	(8,540)	0
Exchange of convertible senior notes for commons stock	0	3,588	0
Cash, Cash Equivalents, Restricted Cash and Restricted Cash Equivalents [Abstract]
Cash and cash equivalents	40,193	31,613	31,331
Restricted cash	260	76,132	52,948
VIE
Supplemental disclosure of non-cash financing/investing activities:
Assets of consolidated VIE	0	1,245,287	0
Liabilities of consolidated VIE	0	(1,231,549)	$ 0
Cash, Cash Equivalents, Restricted Cash and Restricted Cash Equivalents [Abstract]
Cash and cash equivalents	266	0
Restricted cash	$ 260	$ 76,132