Fair Value of Financial Instruments - Fair value, assets measure on recurring basis, unobservable inputs (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Sep. 30, 2017	Sep. 30, 2016
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	$ 67,738	$ 711,503	$ 568,480	$ 1,277,870
Net unrealized gain on mortgage loans	2,185	(8,105)	(64,644)	(39,539)
Net realized gain on mortgage loans	126	9,447	73,077	80,506
Mortgage loan dispositions and payments	(1,355)	(33,108)	(469,328)	(523,457)
Real estate tax advances to borrowers	261	1,161	3,513	6,255
Selling costs on loans held for sale	(406)	0	(1,457)	(1,005)
Accretion of interest on re-performing mortgage loans	0	35	0	107
Transfer of mortgage loans to real estate owned, net	(1,228)	(48,591)	(42,320)	(168,395)
Ending balance	67,321	632,342	67,321	632,342
Net unrealized gain on mortgage loans held at the end of the period	$ 2,237	$ 341	$ (308)	$ 2,581