Other Assets - Real Estate Owned (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($) property	Dec. 31, 2015 USD ($) property	Dec. 31, 2014 USD ($) property
Real Estate Owned Properties [Line Items]
Number of real estate owned properties | property	447	182
Other real estate	$ 80,500	$ 28,000
Number of real estate properties reclassified to REO during period | property	517	186
Transfer from residential whole loans (1)	$ 91,896	$ 30,104	$ 2,904
Residential Whole Loans acquired through foreclosure ordered in lieu	$ 79,300
Number of REO properties sold during period | property	256	63	0
Proceeds from sale of real estate owned	$ 37,900	$ 6,500
Gains/(losses) on sales of real estate owned	3,229	76	$ 0
Adjustments to record at lower of cost or fair value	7,527	3,475
Estimated fair value
Real Estate Owned Properties [Line Items]
Transfer from residential whole loans (1)	91,900	$ 30,100
Mortgage loans in process of foreclosure	501,800
Carrying value
Real Estate Owned Properties [Line Items]
Mortgage loans in process of foreclosure	$ 29,600