Segment Reporting - Results for Reportable Segments (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2020	Dec. 31, 2019
Segment Reporting Revenue Reconciling Item [Line Items]
Total interest income	$ 10,204	$ 9,996	$ 9,780	$ 10,397	$ 10,825	$ 11,027	$ 10,923	$ 10,813	$ 40,377	$ 43,588
Total interest expense	912	1,031	1,157	1,511	1,636	1,680	1,690	1,640	4,611	6,646
Net interest income	9,292	8,965	8,623	8,886	9,189	9,347	9,233	9,173	35,766	36,942
Provision									2,945	2,714
Net interest income after provision for loan and lease losses	8,823	7,919	7,773	8,306	8,473	8,464	8,518	8,773	32,821	34,228
Total non-interest income	1,008	1,375	1,330	1,297	1,396	1,414	1,291	1,265	5,010	5,366
Total non-interest expense	8,477	8,747	8,581	8,494	8,279	8,546	8,504	8,453	34,299	33,782
Income before income taxes									3,532	5,812
Provision for income taxes	309	136	118	262	381	214	300	351	825	1,246
Net income (loss)	1,045	$ 411	$ 404	$ 847	1,209	$ 1,118	$ 1,005	$ 1,234	2,707	4,566
Total assets	890,511				788,738				890,511	788,738
Total investment securities	91,422				108,356				91,422	108,356
Total loans, net	638,374				545,243				638,374	545,243
Goodwill and core deposit intangible, net	8,410				8,825				8,410	8,825
Fixed asset additions									955	3,184
Depreciation and amortization expense									1,684	1,609
Total interest income from external customers									40,377	43,588
Intersegment Eliminations [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total interest income									(2,876)	(5,078)
Total interest expense									(2,876)	(5,078)
Total non-interest income									(4,334)	(6,141)
Total non-interest expense									(582)	(651)
Income before income taxes									(3,752)	(5,490)
Net income (loss)									(3,752)	(5,490)
Total assets	(150,512)				(201,467)				(150,512)	(201,467)
Total loans, net	(47,700)				(95,768)				(47,700)	(95,768)
Investment in subsidiaries	(86,102)				(84,186)				(86,102)	(84,186)
Total interest income from affiliates									(2,876)	(5,078)
FUSB [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total loans, net	585,296				438,710				585,296	438,710
FUSB [Member] | Operating Segments [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total interest income									31,834	30,921
Total interest expense									4,632	6,670
Net interest income									27,202	24,251
Provision									2,206	788
Net interest income after provision for loan and lease losses									24,996	23,463
Total non-interest income									4,531	4,559
Total non-interest expense									25,180	23,065
Income before income taxes									4,347	4,957
Provision for income taxes									930	977
Net income (loss)									3,417	3,980
Total assets	893,430				789,620				893,430	789,620
Total investment securities	91,342				108,276				91,342	108,276
Total loans, net	632,996				534,478				632,996	534,478
Goodwill and core deposit intangible, net	8,410				8,825				8,410	8,825
Fixed asset additions									870	2,948
Depreciation and amortization expense									1,563	1,464
Total interest income from external customers									28,979	25,867
Total interest income from affiliates									2,855	5,054
ALC [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total loans, net	53,078				106,533				53,078	106,533
ALC [Member] | Operating Segments [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total interest income									11,397	17,719
Total interest expense									2,855	5,054
Net interest income									8,542	12,665
Provision									739	1,926
Net interest income after provision for loan and lease losses									7,803	10,739
Total non-interest income									752	909
Total non-interest expense									8,136	9,599
Income before income taxes									419	2,049
Provision for income taxes									125	516
Net income (loss)									294	1,533
Total assets	55,727				110,374				55,727	110,374
Total loans, net	53,078				106,533				53,078	106,533
Fixed asset additions									85	236
Depreciation and amortization expense									121	145
Total interest income from external customers									11,397	17,719
Other Segments | Operating Segments [Member]
Segment Reporting Revenue Reconciling Item [Line Items]
Total interest income									22	26
Net interest income									22	26
Net interest income after provision for loan and lease losses									22	26
Total non-interest income									4,061	6,039
Total non-interest expense									1,565	1,769
Income before income taxes									2,518	4,296
Provision for income taxes									(230)	(247)
Net income (loss)									2,748	4,543
Total assets	91,866				90,211				91,866	90,211
Total investment securities	80				80				80	80
Investment in subsidiaries	$ 86,102				$ 84,186				86,102	84,186
Total interest income from external customers									1	2
Total interest income from affiliates									$ 21	$ 24