Consolidated Balance Sheets (Parenthetical) - USD ($) $ in Thousands		1 Months Ended	12 Months Ended
		Dec. 31, 2019	Dec. 31, 2019	Dec. 31, 2018
Assets
RMBS, pledged assets available-for-sale		$ 2,419,539	$ 2,419,539	$ 1,698,688
Investments in Servicing Related pledged assets at fair value	[1]	$ 291,111	$ 291,111	$ 294,907
Stockholders' Equity
Common stock, par value (in dollars per shares)		$ 0.01	$ 0.01	$ 0.01
Common stock, shares authorized (in shares)		500,000,000	500,000,000	500,000,000
Common stock, shares issued (in shares)		16,660,655	16,660,655	16,652,170
Common stock, shares outstanding (in shares)		16,660,655	16,660,655	16,652,170
Treasury stock (in shares)		235,950	235,950	0
Treasury stock (in dollars per share)		$ 14.75	$ 14.59	$ 0
Series A Preferred Stock [Member]
Stockholders' Equity
Preferred stock, par value (in dollars per shares)		$ 0.01	$ 0.01	$ 0.01
Preferred stock, shares authorized (in shares)		100,000,000	100,000,000	100,000,000
Preferred stock, shares issued (in shares)		2,781,635	2,781,635	2,718,206
Preferred stock, shares outstanding (in shares)		2,781,635	2,781,635	2,718,206
Preferred stock, liquidation preference		$ 69,541	$ 69,541	$ 67,955
Series B Preferred Stock [Member]
Stockholders' Equity
Preferred stock, par value (in dollars per shares)		$ 0.01	$ 0.01	$ 0.01
Preferred stock, shares authorized (in shares)		100,000,000	100,000,000	100,000,000
Preferred stock, shares issued (in shares)		2,000,000	2,000,000	0
Preferred stock, shares outstanding (in shares)		2,000,000	2,000,000	0
Preferred stock, liquidation preference		$ 50,000	$ 50,000	$ 0

[1] Carrying value approximates the fair value of the pools (see Note 9).