Income Taxes (Details) - USD ($) $ in Thousands		3 Months Ended								12 Months Ended
		Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2018		Dec. 31, 2017		Dec. 31, 2016
Components of Income Tax Expense (Benefit) [Abstract]
Current federal income tax expense (benefit)										$ 116		$ (575)		$ 575
Current state income tax expense (benefit)										35		(62)		83
Deferred federal income tax expense (benefit)										1,009		1,046		(198)
Deferred state income tax expense (benefit)										228		192		(2)
Provision for Corporate Business Taxes/Effective Tax Rate		$ (3,137)	$ 729	$ 1,161	$ 2,635	$ 1,143	$ (537)	$ (1,344)	$ 1,339	1,388	[1]	601	[1]	458	[1]
Effective Income Tax Rate Reconciliation, Amount [Abstract]
Computed income tax (benefit) expense at federal rate										8,225		17,015		8,995
State taxes (benefit), net of federal tax, if applicable										222		115		50
Tax benefit due to federal rate change										0		(431)		0
Permanent differences in taxable income from GAAP pre-tax income										0		0		140
Provision to return adjustment										181		(117)		0
REIT income not subject to tax (benefit)										(7,240)		(15,981)		(8,727)
Provision for Corporate Business Taxes/Effective Tax Rate		(3,137)	$ 729	$ 1,161	$ 2,635	1,143	$ (537)	$ (1,344)	$ 1,339	$ 1,388	[1]	$ 601	[1]	$ 458	[1]
Effective Income Tax Rate Reconciliation, Percent [Abstract]
Computed income tax (benefit) expense at federal rate										21.00%		35.00%		35.00%
State taxes (benefit), net of federal tax, if applicable										0.60%		0.20%		0.20%
Tax benefit due to federal rate change										0.00%		(0.90%)		0.00%
Permanent differences in taxable income from GAAP pre-tax income										0.50%		0.00%		0.60%
Provision to return adjustment										0.00%		(0.20%)		0.00%
REIT income not subject to tax (benefit)										(18.60%)		(32.90%)		(34.00%)
Provision for (Benefit from) Corporate Business Taxes/Effective Tax Rate	[1]									3.50%		1.20%		1.80%
Income taxes payable [Abstract]
Federal income taxes payable										$ 116		$ 0
State and local income taxes payable										35		0
Income taxes payable										151		0
Deferred tax (assets) liabilities [Abstract]
Deferred tax - organizational expenses		(4)				(10)				(4)		(10)
Deferred tax - mortgage servicing rights		2,082				909				2,082		909
Deferred tax - net operating loss		0				(56)				0		(56)
Total net deferred tax liabilities		2,078				843				2,078		843
Valuation allowance		0				0				0		0		$ 0
Accrued penalties or Interest		$ 0				$ 0				0		$ 0
Decrease in income tax expense										$ (459)

[1] The provision for income taxes is recorded at the TRS level.