Fair Value - Schedule of Estimated Fair Value and Unpaid Principal Balance, Fair Value Option (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	$ 12,240,962	$ 12,416,542
Total loans at fair value under the fair value option	12,240,962	12,416,542
Total loans at fair value under the fair value option, unpaid principal balance	14,150,322	14,319,767
Debt Instruments At Fair Value Under The Fair Value Option [Abstract]
Mortgage-backed debt at fair value	498,768	514,025
Mortgage-backed debt at fair value, unpaid principal balance	501,603	518,317
HMBS related obligations at fair value	10,289,505	10,509,449
HMBS related obligations at fair value, unpaid principal balance	9,757,690	9,916,383
Total debt instruments at fair value under the fair value option	10,788,273	11,023,474
Total debt instruments at fair value under the fair value option, unpaid principal balance	10,259,293	10,434,700
Reverse Loans
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	10,599,732	10,742,922
Residential loans at fair value, unpaid principal balance	10,139,017	10,218,007
Mortgage Loans Held For Sale
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	1,148,940	1,176,280
Residential loans at fair value, unpaid principal balance	1,098,763	1,148,897
Mortgage Loans Related to Non-Residual Trusts
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	440,219	450,377
Residential loans at fair value, unpaid principal balance	495,041	513,545
Charged-Off Loans
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	52,071	46,963
Residential loans at fair value, unpaid principal balance	$ 2,417,501	$ 2,439,318