Payables and Accrued Liabilities - Schedule of Restructuring Reserve by Segment (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($)
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	$ 4,183
Charges	25,758
Cash payments or other settlements	(17,075)
Balance at end of the year	12,866
Total cumulative charges incurred	37,319
Total expected costs to be incurred	37,608
Servicing [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,174
Charges	11,621
Cash payments or other settlements	(8,019)
Balance at end of the year	4,776
Total cumulative charges incurred	18,083
Total expected costs to be incurred	18,083
Originations [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,663
Charges	3,118
Cash payments or other settlements	(3,498)
Balance at end of the year	1,283
Total cumulative charges incurred	5,726
Total expected costs to be incurred	5,726
Reverse Mortgage [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,346
Charges	5,437
Cash payments or other settlements	(4,286)
Balance at end of the year	2,497
Total cumulative charges incurred	7,077
Total expected costs to be incurred	7,366
Other [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	5,582
Cash payments or other settlements	(1,272)
Balance at end of the year	4,310
Total cumulative charges incurred	6,433
Total expected costs to be incurred	6,433
2015 Actions [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	4,183
Charges	2,212
Cash payments or other settlements	(5,407)
Balance at end of the year	988
Total cumulative charges incurred	13,773
Total expected costs to be incurred	13,773
2015 Actions [Member] | Servicing [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,174
Charges	19
Cash payments or other settlements	(740)
Balance at end of the year	453
Total cumulative charges incurred	6,481
Total expected costs to be incurred	6,481
2015 Actions [Member] | Originations [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,663
Charges	1,982
Cash payments or other settlements	(3,385)
Balance at end of the year	260
Total cumulative charges incurred	4,590
Total expected costs to be incurred	4,590
2015 Actions [Member] | Reverse Mortgage [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	1,346
Charges	211
Cash payments or other settlements	(1,282)
Balance at end of the year	275
Total cumulative charges incurred	1,851
Total expected costs to be incurred	1,851
2015 Actions [Member] | Other [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	0
Cash payments or other settlements	0
Balance at end of the year	0
Total cumulative charges incurred	851
Total expected costs to be incurred	851
2016 Actions [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	23,546
Cash payments or other settlements	(11,668)
Balance at end of the year	11,878
Total cumulative charges incurred	23,546
Total expected costs to be incurred	23,835
2016 Actions [Member] | Servicing [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	11,602
Cash payments or other settlements	(7,279)
Balance at end of the year	4,323
Total cumulative charges incurred	11,602
Total expected costs to be incurred	11,602
2016 Actions [Member] | Originations [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	1,136
Cash payments or other settlements	(113)
Balance at end of the year	1,023
Total cumulative charges incurred	1,136
Total expected costs to be incurred	1,136
2016 Actions [Member] | Reverse Mortgage [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	5,226
Cash payments or other settlements	(3,004)
Balance at end of the year	2,222
Total cumulative charges incurred	5,226
Total expected costs to be incurred	5,515
2016 Actions [Member] | Other [Member]
Restructuring Reserve, [Roll Forward]
Balance at beginning of the year	0
Charges	5,582
Cash payments or other settlements	(1,272)
Balance at end of the year	4,310
Total cumulative charges incurred	5,582
Total expected costs to be incurred	$ 5,582