Segment Reporting (Details) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2020 USD ($)	Mar. 31, 2020 USD ($)	Jun. 30, 2019 USD ($)	Mar. 31, 2019 USD ($)	Jun. 30, 2020 USD ($) item	Jun. 30, 2019 USD ($)	Dec. 31, 2019 USD ($)
Segment Reporting
Number of reportable segments | item					3
Gain (loss) on sale of loans, net	$ 1,451		$ 29,472		$ (26,712)	$ 41,686
Servicing fees, net	1,352		3,536		3,859	6,505
Loss on mortgage servicing rights, net	(8,443)		(9,887)		(26,753)	(15,510)
Real estate services fees, net	293		807		687	1,613
Other revenue	1,289		187		1,352	187
Other operating expense	(14,465)		(21,633)		(45,232)	(43,904)
Other income (expense)	(4,291)		1,472		5,290	851
(Loss) earnings before income taxes	(22,814)		3,954		(87,509)	(8,572)
Income tax expense	15		81		51	167
Net (loss) earnings	(22,829)	$ (64,731)	3,873	$ (12,612)	(87,560)	(8,739)
Assets	2,364,624				2,364,624		$ 3,546,279
Corporate and other
Segment Reporting
Other revenue	1,259		20		1,280	53
Other operating expense	(5,566)		(3,722)		(10,858)	(7,966)
Other income (expense)	(595)		(452)		(1,054)	(909)
(Loss) earnings before income taxes	(4,902)		(4,154)		(10,632)	(8,822)
Assets	33,799				33,799		22,614
Mortgage Lending | Operating segments
Segment Reporting
Gain (loss) on sale of loans, net	1,451		29,472		(26,712)	41,686
Servicing fees, net	1,352		3,536		3,859	6,505
Loss on mortgage servicing rights, net	(8,443)		(9,887)		(26,753)	(15,510)
Other revenue			36			36
Other operating expense	(8,364)		(17,456)		(33,307)	(34,956)
Other income (expense)	287		1,905		2,387	3,320
(Loss) earnings before income taxes	(13,717)		7,606		(80,526)	1,081
Assets	100,640				100,640		888,847
Real Estate Services | Operating segments
Segment Reporting
Real estate services fees, net	293		807		687	1,613
Other operating expense	(356)		(345)		(717)	(732)
(Loss) earnings before income taxes	(63)		462		(30)	881
Assets	508				508		6
Long-term Portfolio | Operating segments
Segment Reporting
Other revenue	30		131		72	98
Other operating expense	(179)		(110)		(350)	(250)
Other income (expense)	(3,983)		19		3,957	(1,560)
(Loss) earnings before income taxes	(4,132)		$ 40		3,679	$ (1,712)
Assets	$ 2,229,677				$ 2,229,677		$ 2,634,812