Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2018		Dec. 31, 2017		Dec. 31, 2016
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	$ 16,958	$ 15,323	$ 12,019	$ 13,415	$ 14,037	$ 11,932	$ 10,002	$ 6,078	$ 57,715		$ 42,049		$ 30,722
Interest expense	10,385	9,257	7,324	7,543	7,062	6,096	4,292	2,431	34,509		19,881		7,808
Net interest income									23,206		22,168		22,914
Servicing fee income	16,574	14,017	11,535	8,650	7,660	6,307	5,493	4,574	50,776		24,034		7,579
Servicing costs	3,528	2,981	2,394	1,712	1,939	1,626	991	1,227	10,615		5,783		2,562
Net servicing income	13,046	11,036	9,141	6,938	5,721	4,681	4,502	3,347	40,161		18,251		5,017
Other income									(14,319)		16,360		4,000
Other operating expenses									9,887		8,164		6,230
Provision for corporate business taxes	(3,137)	729	1,161	2,635	1,143	(537)	(1,344)	1,339	1,388	[1]	601	[1]	458	[1]
Net Income	(38,810)	$ 27,499	$ 13,845	$ 35,239	20,233	$ 6,859	$ (1,665)	$ 22,587	37,773		48,014		25,243
Investments	2,065,017				1,963,718				2,065,017		1,963,718		733,167
Other assets	88,260				86,967				88,260		86,967		59,711
Total Assets	2,153,277				2,050,685				2,153,277		2,050,685		792,878
Debt	1,756,135				1,705,562				1,756,135		1,705,562		617,501
Other liabilities	33,211				22,666				33,211		22,666		19,368
Total Liabilities	1,789,346				1,728,228				1,789,346		1,728,228		636,869
Book value	363,931				322,457				363,931		322,457		156,009
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income									99		523		14,129
Interest expense									2,314		506		1,371
Net interest income									(2,215)		17		12,758
Servicing fee income									50,776		24,034		7,579
Servicing costs									10,615		5,783		2,562
Net servicing income									40,161		18,251		5,017
Other income									(3,573)		15,837		(1,516)
Other operating expenses									0		0		0
Provision for corporate business taxes									1,388		601		458
Net Income									32,985		33,504		15,801
Investments	294,907				122,806				294,907		122,806		61,263
Other assets	17,817				8,281				17,817		8,281		8,826
Total Assets	312,724				131,087				312,724		131,087		70,089
Debt	157,543				39,025				157,543		39,025		22,886
Other liabilities	7,488				6,575				7,488		6,575		2,481
Total Liabilities	165,031				45,600				165,031		45,600		25,367
Book value	147,693				85,487				147,693		85,487		44,722
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income									57,616		41,526		16,593
Interest expense									32,193		19,375		6,437
Net interest income									25,423		22,151		10,156
Servicing fee income									0		0		0
Servicing costs									0		0		0
Net servicing income									0		0		0
Other income									(10,746)		523		5,516
Other operating expenses									0		0		0
Provision for corporate business taxes									0		0		0
Net Income									14,677		22,674		15,672
Investments	1,770,110				1,840,912				1,770,110		1,840,912		671,904
Other assets	38,165				48,631				38,165		48,631		32,495
Total Assets	1,808,275				1,889,543				1,808,275		1,889,543		704,399
Debt	1,598,592				1,666,537				1,598,592		1,666,537		594,615
Other liabilities	10,440				4,385				10,440		4,385		9,490
Total Liabilities	1,609,032				1,670,922				1,609,032		1,670,922		604,105
Book value	199,243				218,621				199,243		218,621		100,294
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income									0		0		0
Interest expense									2		0		0
Net interest income									(2)		0		0
Servicing fee income									0		0		0
Servicing costs									0		0		0
Net servicing income									0		0		0
Other income									0		0		0
Other operating expenses									9,887		8,164		6,230
Provision for corporate business taxes									0		0		0
Net Income									(9,889)		(8,164)		(6,230)
Investments	0				0				0		0		0
Other assets	32,278				30,055				32,278		30,055		18,390
Total Assets	32,278				30,055				32,278		30,055		18,390
Debt	0				0				0		0		0
Other liabilities	15,283				11,706				15,283		11,706		7,397
Total Liabilities	15,283				11,706				15,283		11,706		7,397
Book value	$ 16,995				$ 18,349				$ 16,995		$ 18,349		$ 10,993

[1] The provision for income taxes is recorded at the TRS level.