Consolidated Statements of Cash Flows - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Cash Flows From Operating Activities
Net income	$ 37,773	$ 48,014	$ 25,243
Adjustments to reconcile net income (loss) to net cash provided by (used in) operating activities:
Realized (gain) loss on RMBS, net	8,362	503	(1,399)
Change in fair value of investments in Servicing Related Assets	3,573	(9,159)	3,036
Realized (gain) loss on investments in Excess MSRs, net	0	(6,678)	(1,520)
Realized loss on derivatives, net	5,889	5,554	7,963
Realized (gain) loss on TBA dollar rolls, net	(3,505)	(6,580)	(12,080)
Realized (gain) loss on dollar rolls, net	(1,089)	0	0
Amortization of premiums on investment securities	12,036	8,676	4,115
Amortization of deferred financing costs	205	79	8
LTIP-OP Unit awards	662	607	618
Changes in:
Receivables and other assets	(7,341)	(4,345)	(2,908)
Due to affiliates	(1,032)	1,141	896
Payables for unsettled trades	0	(6,202)	6,202
Accrued expenses and other liabilities and dividends payable	8,105	8,709	4,291
Net cash provided by operating activities	63,638	40,319	34,465
Cash Flows From Investing Activities
Purchase of RMBS	(410,886)	(1,348,134)	(312,095)
Principal paydown of RMBS	170,740	110,409	55,847
Proceeds from sale of RMBS	255,092	62,989	83,735
Principal paydown of Excess MSRs	0	0	7,804
Proceeds from sale of Excess MSRs	0	35,905	42,493
Acquisition of MSRs	(175,674)	(81,611)	(15,395)
Purchase of derivatives	(5,219)	(4,196)	(8,616)
Sale of derivatives	63	163	133
Net cash used in investing activities	(165,884)	(1,224,475)	(146,094)
Cash Flows From Financing Activities
Borrowings under repurchase agreements	6,721,379	4,610,000	2,263,101
Repayments of repurchase agreements	(6,789,324)	(3,538,078)	(2,054,046)
Proceeds from derivative financing	(3,095)	0	0
Proceeds from Federal Home Loan Bank advances	0	0	7,000
Repayments of Federal Home Loan Bank advances	0	0	(69,250)
Proceeds from bank loans	120,317	26,921	13,992
Principal paydown of bank loans	(2,004)	(10,861)	(15,427)
Dividends paid	(37,870)	(24,214)	(15,872)
LTIP-OP Units distributions paid	(367)	(326)	(246)
Conversion of OP units	0	(238)	0
Issuance of common stock, net of offering costs	69,012	81,237	125
Issuance of preferred stock, net of offering costs	7,722	57,917	0
Net cash provided by financing activities	85,770	1,202,358	129,377
Net Increase (Decrease) in Cash, Cash Equivalents and Restricted Cash	(16,476)	18,202	17,748
Cash, Cash Equivalents and Restricted Cash, Beginning of Period	56,495	38,293	20,545
Cash, Cash Equivalents and Restricted Cash, End of Period	40,019	56,495	38,293
Supplemental Disclosure of Cash Flow Information
Cash paid during the period for interest expense	29,821	14,409	6,981
Dividends declared but not paid	$ 11,847	$ 7,273	$ 4,816