Portfolio Loans - Summary of Allowance for Loan Losses by Portfolio Class and Category (Details) - Non-Covered Loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	$ 42,295	$ 38,166	$ 37,565
Provision for loan losses	6,682	9,813	10,764
Charge-offs	(8,594)	(7,976)	(11,507)
Recoveries	2,184	2,292	1,344
Balance, end of year	42,567	42,295	38,166
Commercial and industrial
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	29,039	26,406	26,996
Provision for loan losses	4,801	8,394	8,737
Charge-offs	(6,882)	(6,894)	(9,872)
Recoveries	338	1,133	545
Balance, end of year	27,296	29,039	26,406
CRE - investor owned
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	4,683	3,890	3,420
Provision for loan losses	1,708	709	456
Charge-offs	(551)	0	(117)
Recoveries	95	84	131
Balance, end of year	5,935	4,683	3,890
CRE - owner occupied
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	4,239	3,308	2,890
Provision for loan losses	673	1,216	404
Charge-offs	(58)	(313)	(90)
Recoveries	19	28	104
Balance, end of year	4,873	4,239	3,308
Construction and land development
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	1,987	1,487	1,304
Provision for loan losses	(237)	97	336
Charge-offs	(54)	(56)	(254)
Recoveries	776	459	101
Balance, end of year	2,472	1,987	1,487
Residential real estate
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	1,616	2,237	2,023
Provision for loan losses	(330)	(583)	797
Charge-offs	(667)	(546)	(973)
Recoveries	661	508	390
Balance, end of year	1,280	1,616	2,237
Consumer and other
Allowance for Loan Losses [Roll Forward]
Balance, beginning of year	731	838	932
Provision for loan losses	67	(20)	34
Charge-offs	(382)	(167)	(201)
Recoveries	295	80	73
Balance, end of year	$ 711	$ 731	$ 838