Credit Quality and Nonperforming Assets - Credit Quality Classifications (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019	Mar. 31, 2019
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	$ 1,798,025	$ 1,762,565	$ 1,750,989
Real Estate Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	1,401,188	1,402,114	1,473,850
Real Estate Sector [Member] | 1-4 Family Residential Construction [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	94,578	105,979
Real Estate Sector [Member] | Other construction/land [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	90,602	91,413
Real Estate Sector [Member] | 1-4 Family - Closed-End [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	189,634	200,181
Real Estate Sector [Member] | Equity Lines [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	48,536	49,599
Real Estate Sector [Member] | Multi-Family Residential [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	58,238	54,457
Real Estate Sector [Member] | Commercial Real Estate Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	302,934	343,883
Real Estate Sector [Member] | Commercial Real Estate Non Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	474,926	412,569
Real Estate Sector [Member] | Farmland [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	141,740	144,033
Agricultural Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	49,199	48,036	52,086
Commercial and Industrial Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	111,990	115,532
Mortgage Warehouse Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	228,608	189,103
Consumer Loans Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	7,040	7,780	$ 8,256
Pass [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	1,750,676	1,719,616
Pass [Member] | Real Estate Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	1,369,378	1,374,870
Pass [Member] | Real Estate Sector [Member] | 1-4 Family Residential Construction [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	94,114	105,979
Pass [Member] | Real Estate Sector [Member] | Other construction/land [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	87,861	90,761
Pass [Member] | Real Estate Sector [Member] | 1-4 Family - Closed-End [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	182,832	194,572
Pass [Member] | Real Estate Sector [Member] | Equity Lines [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	42,010	43,111
Pass [Member] | Real Estate Sector [Member] | Multi-Family Residential [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	57,891	54,104
Pass [Member] | Real Estate Sector [Member] | Commercial Real Estate Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	293,360	334,460
Pass [Member] | Real Estate Sector [Member] | Commercial Real Estate Non Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	471,003	409,289
Pass [Member] | Real Estate Sector [Member] | Farmland [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	140,307	142,594
Pass [Member] | Agricultural Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	49,030	47,814
Pass [Member] | Commercial and Industrial Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	97,081	100,584
Pass [Member] | Mortgage Warehouse Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	228,608	189,103
Pass [Member] | Consumer Loans Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	6,579	7,245
Special Mention [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	27,248	24,452
Special Mention [Member] | Real Estate Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	14,020	10,735
Special Mention [Member] | Real Estate Sector [Member] | 1-4 Family Residential Construction [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	464	0
Special Mention [Member] | Real Estate Sector [Member] | Other construction/land [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	2,219	98
Special Mention [Member] | Real Estate Sector [Member] | 1-4 Family - Closed-End [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	3,523	2,425
Special Mention [Member] | Real Estate Sector [Member] | Equity Lines [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	2,084	1,995
Special Mention [Member] | Real Estate Sector [Member] | Multi-Family Residential [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Special Mention [Member] | Real Estate Sector [Member] | Commercial Real Estate Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	3,936	4,005
Special Mention [Member] | Real Estate Sector [Member] | Commercial Real Estate Non Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	747	1,164
Special Mention [Member] | Real Estate Sector [Member] | Farmland [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	1,047	1,048
Special Mention [Member] | Agricultural Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	164	217
Special Mention [Member] | Commercial and Industrial Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	12,994	13,415
Special Mention [Member] | Mortgage Warehouse Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Special Mention [Member] | Consumer Loans Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	70	85
Substandard [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	4,562	4,344
Substandard [Member] | Real Estate Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	4,037	3,763
Substandard [Member] | Real Estate Sector [Member] | 1-4 Family Residential Construction [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Substandard [Member] | Real Estate Sector [Member] | Other construction/land [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Substandard [Member] | Real Estate Sector [Member] | 1-4 Family - Closed-End [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	169	164
Substandard [Member] | Real Estate Sector [Member] | Equity Lines [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	61	72
Substandard [Member] | Real Estate Sector [Member] | Multi-Family Residential [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Substandard [Member] | Real Estate Sector [Member] | Commercial Real Estate Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	3,109	3,384
Substandard [Member] | Real Estate Sector [Member] | Commercial Real Estate Non Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	568	11
Substandard [Member] | Real Estate Sector [Member] | Farmland [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	130	132
Substandard [Member] | Agricultural Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Substandard [Member] | Commercial and Industrial Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	510	556
Substandard [Member] | Mortgage Warehouse Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Substandard [Member] | Consumer Loans Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	15	25
Impaired [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	15,539	14,153
Impaired [Member] | Real Estate Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	13,753	12,746
Impaired [Member] | Real Estate Sector [Member] | 1-4 Family Residential Construction [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Impaired [Member] | Real Estate Sector [Member] | Other construction/land [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	522	554
Impaired [Member] | Real Estate Sector [Member] | 1-4 Family - Closed-End [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	3,110	3,020
Impaired [Member] | Real Estate Sector [Member] | Equity Lines [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	4,381	4,421
Impaired [Member] | Real Estate Sector [Member] | Multi-Family Residential [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	347	353
Impaired [Member] | Real Estate Sector [Member] | Commercial Real Estate Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	2,529	2,034
Impaired [Member] | Real Estate Sector [Member] | Commercial Real Estate Non Owner Occupied [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	2,608	2,105
Impaired [Member] | Real Estate Sector [Member] | Farmland [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	256	259
Impaired [Member] | Agricultural Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	5	5
Impaired [Member] | Commercial and Industrial Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	1,405	977
Impaired [Member] | Mortgage Warehouse Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	0	0
Impaired [Member] | Consumer Loans Sector [Member]
Financing Receivable Recorded Investment [Line Items]
Total gross loans and leases	$ 376	$ 425