Other Assets - Real Estate Owned - Activity (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 56,784	$ 8,857	$ 28,026	$ 5,492
Adjustments to record at lower of cost or fair value	(1,659)	(560)	(4,655)	(1,939)
Transfer from residential whole loans	24,812	5,892	69,803	12,894
Purchases and capital improvements	415	2,052	2,204	2,052
Disposals	(7,163)	(1,716)	(22,189)	(3,974)
Balance at end of period	73,189	14,525	73,189	14,525
Gain recorded on transfer from residential whole loans to real estate owned	$ 845	$ 110	$ 2,500	$ 704