Basis of Presentation and Significant Accounting Policies (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	$ 8,185				$ 29,168				$ 8,185	$ 29,168
Variation margin	3,200								3,200
Realized gain (loss) on RMBS, net [Abstract]
Gain on RMBS									104	213	$ 1,572
Loss on RMBS									(8,466)	(716)	(173)
Net realized loss on RMBS	(2,932)	$ (428)	$ (121)	$ (4,881)	(1)	$ (169)	$ (77)	$ (256)	(8,362)	(503)	1,399
Realized loss on derivatives, net	(3,162)	(707)	(2,033)	13	(1,260)	(1,480)	(1,797)	(1,017)	(5,889)	(5,554)	(7,963)
Unrealized gain on derivatives, net	(30,937)	8,807	6,009	19,626	8,447	1,684	(4,633)	1,082	3,505	6,580	12,080
Realized gain on Excess MSRs, net					0	0	0	6,678	0	6,678	1,520
Unrealized gain on Excess MSRs, net					0	0	0	0	0	0	249
Unrealized gain (loss) on MSRs, net	(21,924)	$ 6,218	$ (365)	$ 12,498	3,688	$ (2,334)	$ (4,507)	$ 12,312	(3,573)	9,159	(3,285)
Total									(14,319)	16,360	$ 4,000
Derivatives [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	687				549				687	549
Repurchase Agreements [Member]
Cash and Cash Equivalents and Restricted Cash [Abstract]
Restricted cash	$ 7,500				$ 28,600				$ 7,500	$ 28,600