Consolidated Statements of Cash Flows - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Cash Flows From Operating Activities
Net income (loss)	$ (43,496)	$ 37,773	$ 48,014
Adjustments to reconcile net income (loss) to net cash provided by (used in) operating activities:
Realized (gain) loss on RMBS, available-for-sale, net	(902)	8,362	503
Unrealized (gain) loss on investments in Servicing Related Assets	106,772	3,573	(9,159)
Realized gain on investments in Excess MSRs, net	0	0	(6,678)
Realized gain on acquired assets, net	(26)	0	0
Realized loss on derivatives, net	12,362	5,889	5,554
Unrealized (gain) loss on derivatives, net	10,867	(3,505)	(6,580)
Realized (gain) on TBA dollar rolls, net	(1,667)	(1,089)	(379)
Amortization of premiums on RMBS, available-for-sale	14,485	12,036	8,676
Amortization of deferred financing costs	652	205	79
LTIP-OP Unit awards	1,023	662	607
Changes in:
Receivables and other assets	(23,075)	(7,341)	(4,345)
Due to affiliates	1,586	(1,032)	1,141
Payables for unsettled trades	0	0	(6,202)
Accrued interest on derivatives	(5,861)	(1,686)	3,875
Dividends payable	(3,079)	4,574	0
Accrued expenses and other liabilities	43	3,531	8,709
Net cash provided by operating activities	69,684	61,952	43,815
Cash Flows From Investing Activities
Purchase of RMBS	(1,088,758)	(410,886)	(1,348,134)
Principal paydown of RMBS	275,086	170,740	110,409
Proceeds from sale of RMBS	141,653	255,092	62,989
Proceeds from sale of Excess MSRs	0	0	35,905
Acquisition of MSRs	(102,976)	(175,674)	(81,611)
Purchase of derivatives	(913)	(3,097)	(2,642)
Proceeds from sale of derivatives	18,793	1	163
Net cash used in investing activities	(757,115)	(163,824)	(1,222,921)
Cash Flows From Financing Activities
Borrowings under repurchase agreements	7,969,930	6,721,379	4,610,000
Repayments of repurchase agreements	(7,230,884)	(6,789,324)	(3,538,078)
Proceeds from derivative financing	(19,090)	(3,469)	(5,050)
Proceeds from bank loans	121,143	120,317	26,921
Principal paydown of bank loans	(112,350)	(2,004)	(10,861)
Dividends paid	(39,227)	(37,870)	(24,214)
LTIP-OP Units distributions paid	(510)	(367)	(326)
Conversion of OP units	(103)	0	(238)
Issuance of common stock, net of offering costs	4,112	69,012	81,237
Issuance of preferred stock, net of offering costs	49,642	7,722	57,917
Repurchase of common stock	(3,543)	0	0
Net cash provided by financing activities	739,120	85,396	1,197,308
Net Increase (Decrease) in Cash, Cash Equivalents and Restricted Cash	51,689	(16,476)	18,202
Cash, Cash Equivalents and Restricted Cash, Beginning of Period	40,019	56,495	38,293
Cash, Cash Equivalents and Restricted Cash, End of Period	91,708	40,019	56,495
Supplemental Disclosure of Cash Flow Information
Cash paid during the period for interest expense	46,533	29,821	14,409
Cash paid during the period for income taxes	256	53	65
Supplemental Schedule of Non-Cash Investing and Financing Activities
Dividends declared but not paid	$ 8,768	$ 11,847	$ 7,273