Income Taxes (Details) - USD ($) $ in Thousands		3 Months Ended		6 Months Ended
		Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021		Jun. 30, 2020		Dec. 31, 2020
Income Taxes [Abstract]
Percentage of taxable income that must be distributed to qualify as a REIT				90.00%
Components of Income Tax Expense (Benefit) [Abstract]
Current federal income tax benefit				$ (128)		$ 0
Deferred federal income tax expense (benefit)				1,595		(14,302)
Deferred state income tax expense (benefit)				166		(1,130)
Provision for (benefit from) Corporate Business Taxes		$ (1,830)	$ (3,278)	1,633	[1]	(15,432)	[1]
Effective Income Tax Rate Reconciliation, Amount [Abstract]
Computed income tax expense (benefit) at federal rate				2,361		(16,160)
State tax expense (benefit), net of federal tax, if applicable				159		(1,130)
Permanent differences in taxable income from GAAP pre-tax income				66		0
REIT income not subject to tax (benefit)				(953)		1,858
Provision for (benefit from) Corporate Business Taxes		(1,830)	(3,278)	$ 1,633	[1]	$ (15,432)	[1]
Effective Income Tax Rate Reconciliation, Percent [Abstract]
Computed income tax expense (benefit) at federal rate				21.00%		21.00%
State tax expense (benefit), net of federal tax, if applicable				1.40%		1.50%
Permanent differences in taxable income from GAAP pre-tax income				0.50%		0.00%
REIT income not subject to tax (benefit)				(8.50%)		(2.40%)
Provision for (benefit from) Effective Tax Rate	[1]			14.40%		20.10%
Income taxes recoverable [Abstract]
Income taxes recoverable		128	$ 0	$ 128		$ 0
Deferred tax assets [Abstract]
Deferred tax - mortgage servicing rights		11,791		11,791				$ 15,176
Deferred tax - net operating loss		7,971		7,971				6,347
Total net deferred tax assets		$ 19,762		$ 19,762				$ 21,523

[1] The provision for income taxes is recorded at the TRS level.