Portfolio Loans - Summary of Allowance for Loan Losses by Portfolio Class and Category (Details) - Non-Covered Loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	$ 38,166	$ 37,565	$ 33,441
Provision (provision reversal)	9,813	10,764	5,551
Losses charged off	(7,976)	(11,507)	(4,335)
Recoveries	2,292	1,344	2,908
Balance, end of year	42,295	38,166	37,565
Commercial and industrial
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	26,406	26,996	22,056
Provision (provision reversal)	8,394	8,737	6,569
Losses charged off	(6,894)	(9,872)	(2,303)
Recoveries	1,133	545	674
Balance, end of year	29,039	26,406	26,996
CRE - investor owned
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	3,890	3,420	3,484
Provision (provision reversal)	709	456	(11)
Losses charged off	0	(117)	(95)
Recoveries	84	131	42
Balance, end of year	4,683	3,890	3,420
CRE - owner occupied
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	3,308	2,890	2,969
Provision (provision reversal)	1,216	404	(1,202)
Losses charged off	(313)	(90)	0
Recoveries	28	104	1,123
Balance, end of year	4,239	3,308	2,890
Construction and land development
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	1,487	1,304	1,704
Provision (provision reversal)	97	336	(1,334)
Losses charged off	(56)	(254)	0
Recoveries	459	101	934
Balance, end of year	1,987	1,487	1,304
Residential real estate
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	2,237	2,023	1,796
Provision (provision reversal)	(583)	797	129
Losses charged off	(546)	(973)	(25)
Recoveries	508	390	123
Balance, end of year	1,616	2,237	2,023
Consumer and other
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	838	932	1,432
Provision (provision reversal)	(20)	34	1,400
Losses charged off	(167)	(201)	(1,912)
Recoveries	80	73	12
Balance, end of year	$ 731	$ 838	$ 932