Portfolio Loans - Summary of Recorded Investment in Impaired Portfolio Loans by Category (Details) - Non-Covered Loans - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	$ 9,680	$ 21,630
Restructured	307	1,360
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	9,987	22,990
Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	4,406	6,073
Restructured	307	0
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	4,713	6,073
Commercial - Investor Owned
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	927	4,597
Restructured	0	590
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	927	5,187
Commercial - Owner Occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	198	519
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	198	519
Construction and land development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	3,444	7,348
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	3,444	7,348
Residential real estate
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	705	3,093
Restructured	0	0
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	705	3,093
Consumer and other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Non-accrual	0	0
Restructured	0	770
Loans over 90 days past due and still accruing interest	0	0
Total Recorded Investment	$ 0	$ 770