Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2019	Mar. 31, 2019	Jun. 30, 2018	Mar. 31, 2018	Jun. 30, 2019		Jun. 30, 2018		Dec. 31, 2018
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	$ 17,216		$ 12,019		$ 34,185		$ 25,434
Interest expense	11,707		7,324		22,451		14,867
Net interest income (expense)	5,509		4,695		11,734		10,567
Servicing fee income	18,362		11,535		35,550		20,185
Servicing costs	4,103		2,394		7,924		4,106
Net servicing income	14,259		9,141		27,626		16,079
Other income (loss)	(48,226)		3,490		(91,149)		30,746
Other operating expenses	3,072		2,320		5,844		4,512
(Benefit from) provision for corporate business taxes	(4,372)		1,161		(9,337)	[1]	3,796	[1]
Net Income (Loss)	(27,158)	$ (21,138)	13,845	$ 35,239	(48,296)		49,084
Investments	2,426,039				2,426,039				$ 2,065,017
Other assets	113,391				113,391				88,260
Total Assets	2,539,430				2,539,430				2,153,277
Debt	2,094,538				2,094,538				1,756,135
Other liabilities	38,812				38,812				33,211
Total Liabilities	2,133,350				2,133,350				1,789,346
Book value	406,080				406,080				363,931
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	214		0		472		0
Interest expense	445		424		1,633		637
Net interest income (expense)	(231)		(424)		(1,161)		(637)
Servicing fee income	18,362		11,535		35,550		20,185
Servicing costs	4,103		2,394		7,924		4,106
Net servicing income	14,259		9,141		27,626		16,079
Other income (loss)	(29,136)		(365)		(54,103)		12,133
Other operating expenses	0		0		0		0
(Benefit from) provision for corporate business taxes	(4,372)		1,161		(9,337)		3,796
Net Income (Loss)	(10,736)		7,191		(18,301)		23,779
Investments	273,699				273,699				294,907
Other assets	41,081				41,081				17,817
Total Assets	314,780				314,780				312,724
Debt	152,027				152,027				157,543
Other liabilities	4,555				4,555				7,488
Total Liabilities	156,582				156,582				165,031
Book value	158,198				158,198				147,693
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	17,002		12,019		33,713		25,434
Interest expense	11,262		6,900		20,818		14,230
Net interest income (expense)	5,740		5,119		12,895		11,204
Servicing fee income	0		0		0		0
Servicing costs	0		0		0		0
Net servicing income	0		0		0		0
Other income (loss)	(19,090)		3,855		(37,046)		18,613
Other operating expenses	0		0		0		0
(Benefit from) provision for corporate business taxes	0		0		0		0
Net Income (Loss)	(13,350)		8,974		(24,151)		29,817
Investments	2,152,340				2,152,340				1,770,110
Other assets	41,568				41,568				38,165
Total Assets	2,193,908				2,193,908				1,808,275
Debt	1,942,511				1,942,511				1,598,592
Other liabilities	19,801				19,801				10,440
Total Liabilities	1,962,312				1,962,312				1,609,032
Book value	231,596				231,596				199,243
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	0		0		0		0
Interest expense	0		0		0		0
Net interest income (expense)	0		0		0		0
Servicing fee income	0		0		0		0
Servicing costs	0		0		0		0
Net servicing income	0		0		0		0
Other income (loss)	0		0		0		0
Other operating expenses	3,072		2,320		5,844		4,512
(Benefit from) provision for corporate business taxes	0		0		0		0
Net Income (Loss)	(3,072)		$ (2,320)		(5,844)		$ (4,512)
Investments	0				0				0
Other assets	30,742				30,742				32,278
Total Assets	30,742				30,742				32,278
Debt	0				0				0
Other liabilities	14,456				14,456				15,283
Total Liabilities	14,456				14,456				15,283
Book value	$ 16,286				$ 16,286				$ 16,995

[1] The provision for income taxes is recorded at the TRS level.