Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended												12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2019		Dec. 31, 2018		Dec. 31, 2017
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	$ 19,770	$ 19,383	$ 17,216	$ 16,969	$ 16,958	$ 15,323	$ 12,019	$ 13,415					$ 73,338		$ 57,715		$ 42,049
Interest expense	13,499	12,635	11,707	10,744	10,385	9,257	7,324	7,543					48,585		34,509		19,881
Net interest income (expense)													24,753		23,206		22,168
Servicing fee income	19,318	18,687	18,362	17,188	16,574	14,017	11,535	8,650					73,555		50,776		24,034
Servicing costs	5,096	4,102	4,103	3,821	3,528	2,981	2,394	1,712					17,122		10,615		5,783
Net servicing income	14,222	14,585	14,259	13,367	13,046	11,036	9,141	6,938					56,433		40,161		18,251
Other income (expense)													(129,073)		(14,319)		16,360
Other operating expenses													12,431		9,887		8,164
(Benefit from) provision for corporate business taxes	(1,842)	(5,643)	(4,372)	(4,965)	(3,137)	729	1,161	2,635					(16,822)	[1]	1,388	[1]	601	[1]
Net Income (Loss)	7,751	$ (2,951)	$ (27,158)	$ (21,138)	(38,810)	$ 27,499	$ 13,845	$ 35,239	$ 20,233	$ 6,859	$ (1,665)	$ 22,587	(43,496)		37,773		48,014
Investments	2,799,471				2,065,017				1,963,718				2,799,471		2,065,017		1,963,718
Other assets	157,081				88,260				86,967				157,081		88,260		86,967
Total Assets	2,956,552				2,153,277				2,050,685				2,956,552		2,153,277		2,050,685
Debt	2,504,627				1,756,135				1,705,562				2,504,627		1,756,135		1,705,562
Other liabilities	40,282				33,211				22,666				40,282		33,211		22,666
Total Liabilities	2,544,909				1,789,346				1,728,228				2,544,909		1,789,346		1,728,228
Book value	411,643				363,931				322,457				411,643		363,931		322,457
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income													2,820		99		523
Interest expense													3,669		2,314		506
Net interest income (expense)													(849)		(2,215)		17
Servicing fee income													73,555		50,776		24,034
Servicing costs													17,122		10,615		5,783
Net servicing income													56,433		40,161		18,251
Other income (expense)													(95,002)		(3,573)		15,837
Other operating expenses													0		0		0
(Benefit from) provision for corporate business taxes													(16,822)		1,388		601
Net Income (Loss)													(22,596)		32,985		33,504
Investments	291,111				294,907				122,806				291,111		294,907		122,806
Other assets	51,729				17,817				8,281				51,729		17,817		8,281
Total Assets	342,840				312,724				131,087				342,840		312,724		131,087
Debt	166,989				157,543				39,025				166,989		157,543		39,025
Other liabilities	9,762				7,488				6,575				9,762		7,488		6,575
Total Liabilities	176,751				165,031				45,600				176,751		165,031		45,600
Book value	166,089				147,693				85,487				166,089		147,693		85,487
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income													70,518		57,616		41,526
Interest expense													44,916		32,193		19,375
Net interest income (expense)													25,602		25,423		22,151
Servicing fee income													0		0		0
Servicing costs													0		0		0
Net servicing income													0		0		0
Other income (expense)													(34,071)		(10,746)		523
Other operating expenses													0		0		0
(Benefit from) provision for corporate business taxes													0		0		0
Net Income (Loss)													(8,469)		14,677		22,674
Investments	2,508,360				1,770,110				1,840,912				2,508,360		1,770,110		1,840,912
Other assets	80,207				38,165				48,631				80,207		38,165		48,631
Total Assets	2,588,567				1,808,275				1,889,543				2,588,567		1,808,275		1,889,543
Debt	2,337,638				1,598,592				1,666,537				2,337,638		1,598,592		1,666,537
Other liabilities	16,503				10,440				4,385				16,503		10,440		4,385
Total Liabilities	2,354,141				1,609,032				1,670,922				2,354,141		1,609,032		1,670,922
Book value	234,426				199,243				218,621				234,426		199,243		218,621
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income													0		0		0
Interest expense													0		2		0
Net interest income (expense)													0		(2)		0
Servicing fee income													0		0		0
Servicing costs													0		0		0
Net servicing income													0		0		0
Other income (expense)													0		0		0
Other operating expenses													12,431		9,887		8,164
(Benefit from) provision for corporate business taxes													0		0		0
Net Income (Loss)													(12,431)		(9,889)		(8,164)
Investments	0				0				0				0		0		0
Other assets	25,145				32,278				30,055				25,145		32,278		30,055
Total Assets	25,145				32,278				30,055				25,145		32,278		30,055
Debt	0				0				0				0		0		0
Other liabilities	14,017				15,283				11,706				14,017		15,283		11,706
Total Liabilities	14,017				15,283				11,706				14,017		15,283		11,706
Book value	$ 11,128				$ 16,995				$ 18,349				$ 11,128		$ 16,995		$ 18,349

[1] The provision for income taxes is recorded at the TRS level.