Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Sep. 30, 2019		Sep. 30, 2018		Dec. 31, 2018
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	$ 19,383			$ 15,323			$ 53,568		$ 40,757
Interest expense	12,635			9,257			35,086		24,124
Net interest income (expense)	6,748			6,066			18,482		16,633
Servicing fee income	18,687			14,017			54,237		34,202
Servicing costs	4,102			2,981			12,026		7,087
Net servicing income	14,585			11,036			42,211		27,115
Other income (expense)	(26,691)			13,890			(117,840)		44,636
Other operating expenses	3,236			2,764			9,080		7,276
(Benefit from) provision for corporate business taxes	(5,643)			729			(14,980)	[1]	4,525	[1]
Net Income (Loss)	(2,951)	$ (27,158)	$ (21,138)	27,499	$ 13,845	$ 35,239	(51,247)		76,583
Investments	2,739,082						2,739,082				$ 2,065,017
Other assets	136,929						136,929				88,260
Total Assets	2,876,011						2,876,011				2,153,277
Debt	2,424,792						2,424,792				1,756,135
Other liabilities	41,612						41,612				33,211
Total Liabilities	2,466,404						2,466,404				1,789,346
Book value	409,607						409,607				363,931
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	380			0			852		0
Interest expense	291			629			1,924		1,266
Net interest income (expense)	89			(629)			(1,072)		(1,266)
Servicing fee income	18,687			14,017			54,237		34,202
Servicing costs	4,102			2,981			12,026		7,087
Net servicing income	14,585			11,036			42,211		27,115
Other income (expense)	(29,423)			6,218			(83,526)		18,351
Other operating expenses	0			0			0		0
(Benefit from) provision for corporate business taxes	(5,643)			729			(14,980)		4,525
Net Income (Loss)	(9,106)			15,896			(27,407)		39,675
Investments	255,845						255,845				294,907
Other assets	57,259						57,259				17,817
Total Assets	313,104						313,104				312,724
Debt	157,951						157,951				157,543
Other liabilities	7,126						7,126				7,488
Total Liabilities	165,077						165,077				165,031
Book value	148,027						148,027				147,693
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	19,003			15,323			52,716		40,757
Interest expense	12,344			8,626			33,162		22,856
Net interest income (expense)	6,659			6,697			19,554		17,901
Servicing fee income	0			0			0		0
Servicing costs	0			0			0		0
Net servicing income	0			0			0		0
Other income (expense)	2,732			7,672			(34,314)		26,285
Other operating expenses	0			0			0		0
(Benefit from) provision for corporate business taxes	0			0			0		0
Net Income (Loss)	9,391			14,369			(14,760)		44,186
Investments	2,483,237						2,483,237				1,770,110
Other assets	58,191						58,191				38,165
Total Assets	2,541,428						2,541,428				1,808,275
Debt	2,266,841						2,266,841				1,598,592
Other liabilities	21,026						21,026				10,440
Total Liabilities	2,287,867						2,287,867				1,609,032
Book value	253,561						253,561				199,243
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	0			0			0		0
Interest expense	0			2			0		2
Net interest income (expense)	0			(2)			0		(2)
Servicing fee income	0			0			0		0
Servicing costs	0			0			0		0
Net servicing income	0			0			0		0
Other income (expense)	0			0			0		0
Other operating expenses	3,236			2,764			9,080		7,276
(Benefit from) provision for corporate business taxes	0			0			0		0
Net Income (Loss)	(3,236)			$ (2,766)			(9,080)		$ (7,278)
Investments	0						0				0
Other assets	21,479						21,479				32,278
Total Assets	21,479						21,479				32,278
Debt	0						0				0
Other liabilities	13,460						13,460				15,283
Total Liabilities	13,460						13,460				15,283
Book value	$ 8,019						$ 8,019				$ 16,995

[1] The provision for income taxes is recorded at the TRS level.