Loans - Additional Information (Details) (USD $)	3 Months Ended	9 Months Ended
	Sep. 30, 2014	Sep. 30, 2014	Dec. 31, 2013
Financing Receivable, Recorded Investment [Line Items]
Transfer of loans from held for investment to held for sale		$ 398,715,000
Covered loans	2,504,622,000	2,504,622,000	2,788,618,000
FDIC indemnification expense	3,200,000	14,700,000
Covered Loans
Financing Receivable, Recorded Investment [Line Items]
30-89 Days Past Due	48,976,000	48,976,000	57,929,000
Total covered loans 90 days or more past due	152,781,000	152,781,000	211,485,000
Covered loan portfolio, current	2,300,000,000	2,300,000,000
Provision for loan losses	3,900,000
Recovery from allowance on losses		18,400,000
Am Trust Bank and Desert Hills Bank
Financing Receivable, Recorded Investment [Line Items]
Delinquent loans selectively extended to certain borrowers, rate reductions, forbearance of arrears, and extension of maturity dates	3,000,000,000	3,000,000,000	3,300,000,000
Financing Receivable Troubled Debt Restructurings Rate Reductions
Financing Receivable, Recorded Investment [Line Items]
Delinquent loans selectively extended to certain borrowers, rate reductions, forbearance of arrears, and extension of maturity dates	37,000,000	37,000,000
Financing Receivable Troubled Debt Restructurings Forbearance Of Arrears
Financing Receivable, Recorded Investment [Line Items]
Delinquent loans selectively extended to certain borrowers, rate reductions, forbearance of arrears, and extension of maturity dates	6,000,000	6,000,000
One-to-Four Family and Commercial and Industrial Loans
Financing Receivable, Recorded Investment [Line Items]
Transfer of loans from held for investment to held for sale		398,700,000
Mortgage Receivable
Financing Receivable, Recorded Investment [Line Items]
Unpaid principal balance of serviced loans	22,100,000,000	22,100,000,000	21,500,000,000
Acquisition, Development and Construction
Financing Receivable, Recorded Investment [Line Items]
Number of loans classified as a non-accrual TDRs		1
Loan classified as non accrual TDRs		935,000
Commercial and Industrial
Financing Receivable, Recorded Investment [Line Items]
Number of loans classified as a non-accrual TDRs		1
Loan classified as non accrual TDRs		499,000
Multi-Family
Financing Receivable, Recorded Investment [Line Items]
Number of loans classified as a non-accrual TDRs		1
Loan classified as non accrual TDRs		316,000
Commercial Real Estate
Financing Receivable, Recorded Investment [Line Items]
Number of loans classified as a non-accrual TDRs		1
Loan classified as non accrual TDRs		2,100,000
Charge-off recorded in connection with CRE loan		334,000
Non-officer directors
Financing Receivable, Recorded Investment [Line Items]
Outstanding loans to non-officer directors	$ 130,100,000	$ 130,100,000	$ 149,400,000