Basis of Presentation and Significant Accounting Policies (Details) - USD ($)	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020	Dec. 31, 2020
Risks and Uncertainties [Abstract]
Percentage of active forbearance loan borrowers	3.00%		3.00%
Percentage of active forbearance loan borrowers paying monthly payment	6.80%		6.80%
Derivatives and Hedging Activities [Abstract]
Net interest income	$ 6,636,000	$ 6,707,000	$ 12,509,000	$ 14,665,000
TBA dollar roll income	3,600,000	1,100,000	6,243,000	1,927,000
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	9,818,000		9,818,000		$ 46,326,000
Variation margin	36,300,000		36,300,000		48,200,000
Realized gain (loss) on RMBS, net [Abstract]
Gain on RMBS	2,820,000	513,000	5,484,000	18,663,000
Loss on RMBS	(1,837,000)	(2,282,000)	(2,407,000)	(37,975,000)
Net realized gain (loss) on RMBS	983,000	(1,769,000)	3,077,000	(19,312,000)
Realized gain (loss) on derivatives, net	(10,139,000)	4,558,000	(14,880,000)	(14,198,000)
Unrealized gain (loss) on derivatives, net	3,548,000	(4,581,000)	(4,511,000)	47,619,000
Realized loss on MSRs, net	0	(11,347,000)	0	(11,347,000)
Unrealized gain (loss) on investments in Servicing Related Assets	(20,501,000)	(17,025,000)	1,963,000	(110,878,000)
Realized gain (loss) on acquired assets, net	29,000	(548,000)	34,000	(502,000)
Total	(26,080,000)	(30,712,000)	(14,317,000)	(108,618,000)
Interest Rate Swaps [Member]
Derivatives and Hedging Activities [Abstract]
Net interest income	891,000	$ 1,500,000	2,200,000	$ 3,000,000.0
Derivatives [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	1,200,000		1,200,000		2,000,000.0
Repurchase Agreements [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	8,600,000		8,600,000		44,400,000
Receivables and Other Assets [Member] | RMBS [Member]
Investments in RMBS [Abstract]
Income receivable	$ 2,400,000		$ 2,400,000		$ 3,000,000.0