Income Taxes - Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Deferred tax assets:
Tax credit carryforwards	$ 37,619	$ 39,560
Allowance for loan and lease losses	11,579	7,784
Net operating loss carryforwards	83	5,046
Mark to market on loans held for sale	10,501	1,780
Stock-based compensation expense	4,918	3,004
Goodwill and intangibles	720	720
Accrued expenses	1,372	430
Net unrealized losses on securities available for sale	0	529
Operating lease liabilities	629	0
Other	977	1,573
Total deferred tax assets	68,398	60,426
Deferred tax liabilities:
Investment in joint venture	15,538	16,596
Unguaranteed loan discount	13,076	8,535
Premises and equipment	45,291	40,032
Deferred loan fees and costs, net	1,417	987
Net unrealized gains on securities available for sale	3,702	0
Operating lease right-of-use assets	584	0
Other	538	326
Total deferred tax liabilities	80,146	66,476
Net deferred tax liability	$ 11,748	$ 6,050