Fair Value - Schedule of Estimated Fair Value and Unpaid Principal Balance, Fair Value Option (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Option, Quantitative Disclosures [Line Items]
Residential loans at fair value	$ 12,416,542	$ 12,673,439
Total loans at fair value under the fair value option	12,416,542	12,673,439
Total loans at fair value under the fair value option, unpaid principal balance	14,319,767	14,941,165
Mortgage-backed debt at fair value	514,025	582,340
Mortgage-backed debt at fair value, unpaid principal balance	518,317	585,839
HMBS related obligations at fair value	10,509,449	10,647,382
HMBS related obligations at fair value, unpaid principal balance	9,916,383	10,012,283
Total debt instruments at fair value under the fair value option	11,023,474	11,229,722
Total debt instruments at fair value under the fair value option, unpaid principal balance	10,434,700	10,598,122
Reverse Loans [Member]
Fair Value, Option, Quantitative Disclosures [Line Items]
Residential loans at fair value	10,742,922	10,763,816
Residential loans at fair value, unpaid principal balance	10,218,007	10,187,521
Mortgage Loans Held For Sale [Member]
Fair Value, Option, Quantitative Disclosures [Line Items]
Residential loans at fair value	1,176,280	1,334,300
Residential loans at fair value, unpaid principal balance	1,148,897	1,285,582
Mortgage Loans Related To Non Residual Trusts [Member]
Fair Value, Option, Quantitative Disclosures [Line Items]
Residential loans at fair value	450,377	526,016
Residential loans at fair value, unpaid principal balance	513,545	580,695
Charged-off Loans [Member]
Fair Value, Option, Quantitative Disclosures [Line Items]
Residential loans at fair value	46,963	49,307
Residential loans at fair value, unpaid principal balance	$ 2,439,318	$ 2,887,367