Fair Value - Schedule of Estimated Fair Value and Unpaid Principal Balance, Fair Value Option (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	$ 11,905,869	$ 12,416,542
Total loans at fair value under the fair value option	11,905,869	12,416,542
Total loans at fair value under the fair value option, unpaid principal balance	13,848,818	14,319,767
Debt Instruments At Fair Value Under The Fair Value Option [Abstract]
Mortgage-backed debt at fair value	470,600	514,025
Mortgage-backed debt at fair value, unpaid principal balance	474,980	518,317
HMBS related obligations at fair value	9,986,409	10,509,449
HMBS related obligations at fair value, unpaid principal balance	9,481,428	9,916,383
Total debt instruments at fair value under the fair value option	10,457,009	11,023,474
Total debt instruments at fair value under the fair value option, unpaid principal balance	9,956,408	10,434,700
Reverse Loans
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	10,440,669	10,742,922
Residential loans at fair value, unpaid principal balance	10,018,774	10,218,007
Mortgage Loans Held For Sale
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	1,009,568	1,176,280
Residential loans at fair value, unpaid principal balance	969,551	1,148,897
Mortgage Loans Related to Non-Residual Trusts
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	406,006	450,377
Residential loans at fair value, unpaid principal balance	469,429	513,545
Charged-Off Loans
Loans At Fair Value Under The Fair Value Option [Abstract]
Residential loans at fair value	49,626	46,963
Residential loans at fair value, unpaid principal balance	$ 2,391,064	$ 2,439,318