Other Assets and Liabilities - Accrued Expenses and Other Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Deferred Costs, Capitalized, Prepaid, and Other Assets Disclosure [Abstract]
Guarantee obligations	$ 21,668	$ 22,704
Accrued compensation	18,830	17,527
Margin payable	12,783	6,415
Residential loan and MSR repurchase reserve	5,432	6,403
Accrued operating expenses	4,493	1,845
Restructuring liabilities	2,297	0
Legal reserve	2,000	2,000
Current accounts payable	1,151	4,764
Deferred tax liability	898	0
Other	2,876	8,239
Total Other Liabilities	$ 72,428	$ 69,897