Payables and Accrued Liabilities - Schedule of Exit Liability by Segment (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2017 USD ($)
Exit Liability [Roll Forward]
Balance at January 1, 2017	$ 12,866
Charges	1,871
Cash payments or other settlements	(10,191)
Balance at March 31, 2017	4,546
Total cumulative charges incurred	39,190
Total expected costs to be incurred (2)	44,311
Servicing
Exit Liability [Roll Forward]
Balance at January 1, 2017	4,776
Charges	823
Cash payments or other settlements	(3,551)
Balance at March 31, 2017	2,048
Total cumulative charges incurred	18,906
Total expected costs to be incurred (2)	22,138
Originations
Exit Liability [Roll Forward]
Balance at January 1, 2017	1,283
Charges	433
Cash payments or other settlements	(1,081)
Balance at March 31, 2017	635
Total cumulative charges incurred	6,159
Total expected costs to be incurred (2)	6,189
Reverse Mortgage
Exit Liability [Roll Forward]
Balance at January 1, 2017	2,497
Charges	678
Cash payments or other settlements	(1,905)
Balance at March 31, 2017	1,270
Total cumulative charges incurred	7,755
Total expected costs to be incurred (2)	8,738
Other
Exit Liability [Roll Forward]
Balance at January 1, 2017	4,310
Charges	(63)
Cash payments or other settlements	(3,654)
Balance at March 31, 2017	593
Total cumulative charges incurred	6,370
Total expected costs to be incurred (2)	7,246
2015 Actions
Exit Liability [Roll Forward]
Balance at January 1, 2017	988
Charges	(47)
Cash payments or other settlements	(217)
Balance at March 31, 2017	724
Total cumulative charges incurred	13,726
Total expected costs to be incurred (2)	13,726
2015 Actions | Servicing
Exit Liability [Roll Forward]
Balance at January 1, 2017	453
Charges	(57)
Cash payments or other settlements	(66)
Balance at March 31, 2017	330
Total cumulative charges incurred	6,424
Total expected costs to be incurred (2)	6,424
2015 Actions | Originations
Exit Liability [Roll Forward]
Balance at January 1, 2017	260
Charges	7
Cash payments or other settlements	(47)
Balance at March 31, 2017	220
Total cumulative charges incurred	4,597
Total expected costs to be incurred (2)	4,597
2015 Actions | Reverse Mortgage
Exit Liability [Roll Forward]
Balance at January 1, 2017	275
Charges	3
Cash payments or other settlements	(104)
Balance at March 31, 2017	174
Total cumulative charges incurred	1,854
Total expected costs to be incurred (2)	1,854
2015 Actions | Other
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	0
Cash payments or other settlements	0
Balance at March 31, 2017	0
Total cumulative charges incurred	851
Total expected costs to be incurred (2)	851
2016 Actions
Exit Liability [Roll Forward]
Balance at January 1, 2017	11,878
Charges	(339)
Cash payments or other settlements	(9,368)
Balance at March 31, 2017	2,171
Total cumulative charges incurred	23,207
Total expected costs to be incurred (2)	23,207
2016 Actions | Servicing
Exit Liability [Roll Forward]
Balance at January 1, 2017	4,323
Charges	(56)
Cash payments or other settlements	(3,046)
Balance at March 31, 2017	1,221
Total cumulative charges incurred	11,546
Total expected costs to be incurred (2)	11,546
2016 Actions | Originations
Exit Liability [Roll Forward]
Balance at January 1, 2017	1,023
Charges	(80)
Cash payments or other settlements	(872)
Balance at March 31, 2017	71
Total cumulative charges incurred	1,056
Total expected costs to be incurred (2)	1,056
2016 Actions | Reverse Mortgage
Exit Liability [Roll Forward]
Balance at January 1, 2017	2,222
Charges	(140)
Cash payments or other settlements	(1,796)
Balance at March 31, 2017	286
Total cumulative charges incurred	5,086
Total expected costs to be incurred (2)	5,086
2016 Actions | Other
Exit Liability [Roll Forward]
Balance at January 1, 2017	4,310
Charges	(63)
Cash payments or other settlements	(3,654)
Balance at March 31, 2017	593
Total cumulative charges incurred	5,519
Total expected costs to be incurred (2)	5,519
2017 Actions
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	2,257
Cash payments or other settlements	(606)
Balance at March 31, 2017	1,651
Total cumulative charges incurred	2,257
Total expected costs to be incurred (2)	7,378
2017 Actions | Servicing
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	936
Cash payments or other settlements	(439)
Balance at March 31, 2017	497
Total cumulative charges incurred	936
Total expected costs to be incurred (2)	4,168
2017 Actions | Originations
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	506
Cash payments or other settlements	(162)
Balance at March 31, 2017	344
Total cumulative charges incurred	506
Total expected costs to be incurred (2)	536
2017 Actions | Reverse Mortgage
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	815
Cash payments or other settlements	(5)
Balance at March 31, 2017	810
Total cumulative charges incurred	815
Total expected costs to be incurred (2)	1,798
2017 Actions | Other
Exit Liability [Roll Forward]
Balance at January 1, 2017	0
Charges	0
Cash payments or other settlements	0
Balance at March 31, 2017	0
Total cumulative charges incurred	0
Total expected costs to be incurred (2)	$ 876