Income Taxes (Details) - USD ($) $ in Thousands		3 Months Ended								12 Months Ended
		Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019		Dec. 31, 2018		Dec. 31, 2017
Income Taxes [Abstract]
Percentage of taxable income that must be distributed to qualify as a REIT										90.00%
Components of Income Tax Expense (Benefit) [Abstract]
Current federal income tax expense (benefit)										$ 0		$ 116		$ (575)
Current state income tax expense (benefit)										0		35		(62)
Deferred federal income tax expense (benefit)										(15,360)		1,009		1,046
Deferred state income tax expense (benefit)										(1,462)		228		192
Provision for (Benefit from) Corporate Business Taxes		$ (1,842)	$ (5,643)	$ (4,372)	$ (4,965)	$ (3,137)	$ 729	$ 1,161	$ 2,635	(16,822)	[1]	1,388	[1]	601	[1]
Effective Income Tax Rate Reconciliation, Amount [Abstract]
Computed income tax (benefit) expense at federal rate										(12,665)		8,225		17,015
State taxes (benefit), net of federal tax, if applicable										(1,213)		222		115
Tax benefit due to federal rate change										0		0		(431)
Tax provision due to State tax rate change										(249)		0		0
Provision to return adjustment										5		181		(117)
REIT income not subject to tax (benefit)										(2,700)		(7,240)		(15,981)
Provision for (Benefit from) Corporate Business Taxes		(1,842)	$ (5,643)	$ (4,372)	$ (4,965)	(3,137)	$ 729	$ 1,161	$ 2,635	$ (16,822)	[1]	$ 1,388	[1]	$ 601	[1]
Effective Income Tax Rate Reconciliation, Percent [Abstract]
Computed income tax (benefit) expense at federal rate										21.00%		21.00%		35.00%
State taxes (benefit), net of federal tax, if applicable										2.00%		0.60%		0.20%
Tax benefit due to federal rate change										0.000		0.000		(0.009)
Tax provision due to State tax rate change										0.004		0.000		0.000
Provision to return adjustment										0.00%		0.50%		(0.20%)
REIT income not subject to tax (benefit)										4.50%		(18.60%)		(32.90%)
Provision for (Benefit from) Corporate Business Taxes/Effective Tax Rate	[1]									27.90%		3.50%		1.20%
Income taxes payable [Abstract]
Federal income taxes payable										$ 0		$ 116		$ 0
State and local income taxes payable										0		35		0
Income taxes payable										0		151		0
Deferred tax (assets) liabilities [Abstract]
Deferred tax - organizational expenses		0				(4)				0		(4)		(10)
Deferred tax - mortgage servicing rights		(13,045)				2,082				(13,045)		2,082		909
Deferred tax - net operating loss		(1,699)				0				(1,699)		0		(56)
Total net deferred tax assets		(14,744)								(14,744)
Total net deferred tax liabilities						2,078						2,078		843
Valuation allowance		0				0				0		0		0
Accrued penalties or Interest		$ 0				$ 0				$ 0		$ 0
Decrease in income tax expense														$ (459)

[1] The provision for income taxes is recorded at the TRS level.