Loans - Covered Loans Acquired in AmTrust and Desert Hills Acquisitions (Details) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014	Dec. 31, 2013
Accounts, Notes, Loans and Financing Receivable [Line Items]
Covered loans	$ 2,504,622	$ 2,788,618
Percent of Covered Loans	100.00%
One-to-four family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Covered loans	2,282,064
Percent of Covered Loans	91.10%
Other loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Covered loans	$ 222,558
Percent of Covered Loans	8.90%