Residential Loans - Summary of Activity in Allowance for Losses on Residential Loans (Details) - USD ($) $ in Thousands	3 Months Ended				12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Allowance for Loan and Lease Losses [Roll Forward]
Reversal of provision for loan losses	$ 0	$ 859	$ 6,532	$ (289)	$ 7,102	$ 355	$ (961)
Held-for sale residential loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance at beginning of period				$ 0	0	21,338	25,427
Charge-offs, net					0	0	(4,966)
Reversal of provision for loan losses					0	0	877
Other adjustments					0	(21,338)	0
Balance at End of Period	$ 0				$ 0	$ 0	$ 21,338