Income Taxes (Details) - USD ($) $ in Thousands		3 Months Ended		9 Months Ended
		Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019		Sep. 30, 2018		Dec. 31, 2018
Income Taxes [Abstract]
Percentage of taxable income that must be distributed to qualify as a REIT				90.00%
Components of Income Tax Expense (Benefit) [Abstract]
Current federal income tax expense (benefit)				$ 0		$ 156
Current state income tax expense (benefit)				0		47
Deferred federal income tax expense (benefit)				(12,222)		3,526
Deferred state income tax expense (benefit)				(2,758)		796
Provision for (Benefit from) Corporate Business Taxes		$ (5,643)	$ 729	(14,980)	[1]	4,525	[1]
Effective Income Tax Rate Reconciliation, Amount [Abstract]
Computed income tax (benefit) expense at federal rate				(13,906)		17,035
State taxes (benefit), net of federal tax, if applicable				(2,757)		832
Provision to return adjustment				0		4
REIT income not subject to tax (benefit)				1,683		(13,346)
Provision for (Benefit from) Corporate Business Taxes		(5,643)	$ 729	$ (14,980)	[1]	$ 4,525	[1]
Effective Income Tax Rate Reconciliation, Percent [Abstract]
Computed income tax (benefit) expense at federal rate				21.00%		21.00%
State taxes (benefit), net of federal tax, if applicable				4.20%		1.00%
Provision to return adjustment				0.00%		0.00%
REIT income not subject to tax (benefit)				(2.60%)		(16.40%)
Provision for (Benefit from) Corporate Business Taxes/Effective Tax Rate	[1]			22.60%		5.60%
Income taxes payable [Abstract]
Federal income taxes payable				$ 0		$ 156
State and local income taxes payable				0		47
Income taxes payable				0		$ 203
Deferred tax (assets) liabilities [Abstract]
Deferred tax - organizational expenses		0		0				$ (4)
Deferred tax - mortgage servicing rights		(12,651)		(12,651)				2,082
Deferred tax - net operating loss		(250)		(250)				0
Total net deferred tax assets		(12,901)		(12,901)
Total net deferred tax liabilities								2,078
Valuation allowance		$ 0		$ 0				$ 0

[1] The provision for income taxes is recorded at the TRS level.