Consolidated Statements of Income (Loss) - USD ($) $ in Thousands	3 Months Ended												12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2019		Dec. 31, 2018		Dec. 31, 2017
Income
Interest income	$ 19,770	$ 19,383	$ 17,216	$ 16,969	$ 16,958	$ 15,323	$ 12,019	$ 13,415					$ 73,338		$ 57,715		$ 42,049
Interest expense	13,499	12,635	11,707	10,744	10,385	9,257	7,324	7,543					48,585		34,509		19,881
Net interest income	6,271	6,748	5,509	6,225	6,573	6,066	4,695	5,872					24,753		23,206		22,168
Servicing fee income	19,318	18,687	18,362	17,188	16,574	14,017	11,535	8,650					73,555		50,776		24,034
Servicing costs	5,096	4,102	4,103	3,821	3,528	2,981	2,394	1,712					17,122		10,615		5,783
Net servicing income	14,222	14,585	14,259	13,367	13,046	11,036	9,141	6,938					56,433		40,161		18,251
Other income (loss)
Realized gain (loss) on RMBS, available-for-sale, net	627	275	0	0	(2,932)	(428)	(121)	(4,881)					902		(8,362)		(503)
Realized gain on investments in Excess MSRs, net													0		0		6,678
Realized loss on derivatives, net	(17,148)	12,627	(365)	(7,476)	(3,162)	(707)	(2,033)	13					(12,362)		(5,889)		(5,554)
Realized gain (loss) on acquired assets, net	(28)	54	0	0									26		0		0
Unrealized gain (loss) on derivatives, net	3,357	(2,133)	(3,819)	(8,272)	(30,937)	8,807	6,009	19,626					(10,867)		3,505		6,580
Unrealized gain (loss) on investments in Servicing Related Assets	1,959	(37,514)	(44,042)	(27,175)	(21,924)	6,218	(365)	12,498					(106,772)		(3,573)		9,159
Total Income (Loss)	9,260	(5,358)	(28,458)	(23,331)	(39,336)	30,992	17,326	40,066					(47,887)		49,048		56,779
Expenses
General and administrative expense	1,352	1,194	1,138	963	962	1,165	937	877					4,647		3,941		3,817
Management fee to affiliate	1,999	2,042	1,934	1,809	1,649	1,599	1,383	1,315					7,784		5,946		4,347
Total Expenses	3,351	3,236	3,072	2,772	2,611	2,764	2,320	2,192					12,431		9,887		8,164
Income (Loss) Before Income Taxes	5,909	(8,594)	(31,530)	(26,103)	(41,947)	28,228	15,006	37,874					(60,318)		39,161		48,615
Provision for (Benefit from) corporate business taxes	(1,842)	(5,643)	(4,372)	(4,965)	(3,137)	729	1,161	2,635					(16,822)	[1]	1,388	[1]	601	[1]
Net Income (Loss)	7,751	(2,951)	(27,158)	(21,138)	(38,810)	27,499	13,845	35,239	$ 20,233	$ 6,859	$ (1,665)	$ 22,587	(43,496)		37,773		48,014
Net (income) loss allocated to noncontrolling interests in Operating Partnership	(127)	43	438	349	505	(364)	(173)	(456)					703		(488)		(655)
Dividends on preferred stock	2,460	2,459	2,593	1,841	1,395	1,372	1,317	1,213					9,353		5,297		1,833
Net Income (Loss) Applicable to Common Stockholders	$ 5,164	$ (5,367)	$ (29,313)	$ (22,630)	$ (39,700)	$ 25,763	$ 12,355	$ 33,570					$ (52,146)		$ 31,988		$ 45,526
Net Income (Loss) Per Share of Common Stock
Basic (in dollars per share)	$ 0.31	$ (0.32)	$ (1.75)	$ (1.36)	$ (2.42)	$ 1.62	$ 0.91	$ 2.64					$ (3.11)		$ 2.18		$ 3.98
Diluted (in dollars per share)	$ 0.31	$ (0.32)	$ (1.75)	$ (1.36)	$ (2.42)	$ 1.62	$ 0.91	$ 2.64					$ (3.11)		$ 2.18		$ 3.98
Weighted Average Number of Shares of Common Stock Outstanding
Basic (in shares)	16,797,523	16,883,816	16,776,472	16,646,114	16,382,914	15,864,774	13,616,461	12,713,265					16,775,113		14,649,242		11,443,493
Diluted (in shares)	16,810,312	16,896,605	16,789,261	16,654,370	16,391,170	15,873,030	13,624,676	12,721,464					16,787,902		14,657,498		11,450,831

[1] The provision for income taxes is recorded at the TRS level.