Segment Reporting (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2020 USD ($) item	Mar. 31, 2019 USD ($)	Dec. 31, 2019 USD ($)
Segment Reporting
Number of reportable segments | item	3
(Loss) gain on sale of loans, net	$ (28,162)	$ 12,214
Servicing fees, net	2,507	2,969
Loss on mortgage servicing rights, net	(18,310)	(5,623)
Real estate services fees, net	393	806
Other revenue	63
Other operating expense	(30,767)	(22,270)
Other income (expense)	9,581	(622)
Loss before income taxes	(64,695)	(12,526)
Income tax expense	36	86
Net loss	(64,731)	(12,612)
Assets	2,922,098		$ 3,546,279
Corporate and other
Segment Reporting
Other revenue	21	32
Other operating expense	(5,291)	(4,245)
Other income (expense)	(459)	(457)
Loss before income taxes	(5,729)	(4,670)
Assets	31,916		22,614
Mortgage Lending | Operating segments
Segment Reporting
(Loss) gain on sale of loans, net	(28,162)	12,214
Servicing fees, net	2,507	2,969
Loss on mortgage servicing rights, net	(18,310)	(5,623)
Other operating expense	(24,944)	(17,500)
Other income (expense)	2,100	1,415
Loss before income taxes	(66,809)	(6,525)
Assets	636,798		888,847
Real Estate Services | Operating segments
Segment Reporting
Real estate services fees, net	393	806
Other operating expense	(361)	(387)
Loss before income taxes	32	419
Assets	2		6
Long-term Portfolio | Operating segments
Segment Reporting
Other revenue	42	(32)
Other operating expense	(171)	(138)
Other income (expense)	7,940	(1,580)
Loss before income taxes	7,811	$ (1,750)
Assets	$ 2,253,382		$ 2,634,812