Basis of Presentation and Significant Accounting Policies (Details) - USD ($)		3 Months Ended
		Mar. 31, 2022	Mar. 31, 2021	Dec. 31, 2021
Risks and Uncertainties [Abstract]
Percentage of active forbearance loan borrowers		1.30%
Percentage of active forbearance loan borrowers paying monthly payment		10.60%
Investments in RMBS [Abstract]
Sale of RMBS, settled after period end		$ (46,801,000)	$ (16,702,000)
Investments in MSRs [Abstract]
Net interest income		3,879,000	1,847,000
Servicing fee income		13,116,000	13,540,000
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash		26,974,000		$ 12,861,000
Variation margin		64,300,000		45,600,000
Realized gain (loss) on RMBS, net [Abstract]
Gain on RMBS		50,000	2,664,000
Loss on RMBS		(13,272,000)	(570,000)
Net realized gain (loss) on RMBS	[1]	(13,222,000)	2,094,000
MSRs [Member]
Investments in MSRs [Abstract]
Net interest income		192,000	230,000
Servicing fee income		494,000	$ 321,000
Reserves for unrecoverable advances		0		0
Derivatives [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash		0		664,000
Repurchase Agreements [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash		27,000,000.0		12,200,000
Receivables and Other Assets [Member] | RMBS [Member]
Investments in RMBS [Abstract]
Income receivable		$ 2,100,000		$ 2,300,000

[1] Reclassified from accumulated other comprehensive income into earnings.