Segment Reporting - Statement of Operations Items (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2017 USD ($) item	Mar. 31, 2016 USD ($)
Segment Reporting
Number of reportable segments | item	3
Gain on sale of loans, net	$ 37,319	$ 53,869
Real estate services fees, net	1,633	2,100
Servicing income, net	7,320	2,088
Loss on mortgage servicing rights, net	(977)	(10,910)
Other revenue	47	152
Accretion of contingent consideration	(845)	(1,895)
Change in fair value of contingent consideration	(539)	(2,942)
Change in fair value of long-term debt	(2,497)
Other (expense) income	(36,408)	(41,046)
Earnings before income taxes	5,053	1,416
Income tax expense	426	435
Net earnings	4,627	981
Corporate and Other.
Segment Reporting
Other revenue	(28)	36
Other (expense) income	(4,859)	(2,979)
Earnings before income taxes	(4,887)	(2,943)
Mortgage Lending | Operating segments
Segment Reporting
Gain on sale of loans, net	37,319	53,869
Servicing income, net	7,320	2,088
Loss on mortgage servicing rights, net	(977)	(10,910)
Other revenue	14	49
Accretion of contingent consideration	(845)	(1,895)
Change in fair value of contingent consideration	(539)	(2,942)
Other (expense) income	(37,678)	(37,140)
Earnings before income taxes	4,614	3,119
Real Estate Services | Operating segments
Segment Reporting
Real estate services fees, net	1,633	2,100
Other (expense) income	(995)	(1,566)
Earnings before income taxes	638	534
Long-term Portfolio | Operating segments
Segment Reporting
Other revenue	61	67
Change in fair value of long-term debt	(2,497)
Other (expense) income	7,124	639
Earnings before income taxes	$ 4,688	$ 706