Income Taxes - Schedule of Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Jun. 30, 2019
Deferred tax assets:
Alternative minimum tax credit	$ 0	$ 4,799
Allowance for loan losses	6,456	4,685
Deferred compensation and post-retirement benefits	8,637	8,988
Accrued vacation and sick leave	18	18
Impairments on real estate owned	198	461
Other than temporary impairment on investments	2,207	2,232
Net operating loss carryforward	4,513	5,092
Discount from business combination	2,192	2,373
Stock compensation plans	2,279	2,162
Other	1,256	1,140
Total gross deferred tax assets	27,756	31,950
Deferred tax (liabilities):
Depreciable basis of fixed assets	(6,017)	(1,089)
Deferred loan fees	(603)	(520)
FHLB stock, book basis in excess of tax	(89)	(89)
Unrealized gain on securities available for sale	(602)	(219)
Other	(4,111)	(3,510)
Total gross deferred tax liabilities	(11,422)	(5,427)
Net deferred tax assets	$ 16,334	$ 26,523