Loans - Schedule of Ending Balances of Loans and the Related Allowance by Segment and Class (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2017
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	$ 28,072	$ 21,429	$ 21,060	$ 21,151
Allowance for loan losses, Loans individually evaluated for impairment	1,258	120
Allowance for loan losses, Loans collectively evaluated	26,632	21,108
Allowance for loan losses, Total	28,072	21,429
Total loans receivable, PCI	2,756,138	2,688,436
Total loans receivable, Loans individually evaluated for impairment	14,486	15,743
Total loans receivable, Loans collectively evaluated	2,741,652	2,672,662
Total loans	2,768,930	2,705,186
Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	182	201	483	727
Total loans receivable, PCI	12,792	16,781
Retail Consumer Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	6,906	6,419	7,527	8,585
Total loans	907,342	1,162,464
Retail Consumer Loans | One-to-four family
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	52	74
Allowance for loan losses, Loans collectively evaluated	2,400	2,375
Allowance for loan losses, Total	2,469	2,511
Total loans receivable, PCI	469,220	654,723
Total loans receivable, Loans individually evaluated for impairment	4,304	5,318
Total loans receivable, Loans collectively evaluated	464,916	649,405
Total loans	473,693	660,591
Retail Consumer Loans | One-to-four family | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	17	62
Total loans receivable, PCI	4,473	5,868
Retail Consumer Loans | HELOCs - originated
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	7
Allowance for loan losses, Loans collectively evaluated	1,344	1,023
Allowance for loan losses, Total	1,344	1,030
Total loans receivable, PCI	137,447	139,210
Total loans receivable, Loans individually evaluated for impairment	0	7
Total loans receivable, Loans collectively evaluated	137,447	139,203
Total loans	137,447	139,435
Retail Consumer Loans | HELOCs - originated | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	225
Retail Consumer Loans | HELOCs - purchased
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	0
Allowance for loan losses, Loans collectively evaluated	430	518
Allowance for loan losses, Total	430	518
Total loans receivable, PCI	71,781	116,972
Total loans receivable, Loans individually evaluated for impairment	0	0
Total loans receivable, Loans collectively evaluated	71,781	116,972
Total loans	71,781	116,972
Retail Consumer Loans | HELOCs - purchased | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	0
Retail Consumer Loans | Construction and land/lots
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	0
Allowance for loan losses, Loans collectively evaluated	1,409	1,265
Allowance for loan losses, Total	1,442	1,265
Total loans receivable, PCI	81,514	80,230
Total loans receivable, Loans individually evaluated for impairment	296	323
Total loans receivable, Loans collectively evaluated	81,218	79,907
Total loans	81,859	80,602
Retail Consumer Loans | Construction and land/lots | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	33	0
Total loans receivable, PCI	345	372
Retail Consumer Loans | Indirect auto finance
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	0
Allowance for loan losses, Loans collectively evaluated	1,136	927
Allowance for loan losses, Total	1,136	927
Total loans receivable, PCI	132,303	153,448
Total loans receivable, Loans individually evaluated for impairment	10	0
Total loans receivable, Loans collectively evaluated	132,293	153,448
Total loans	132,303	153,448
Retail Consumer Loans | Indirect auto finance | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	0
Retail Consumer Loans | Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	4
Allowance for loan losses, Loans collectively evaluated	135	226
Allowance for loan losses, Total	135	230
Total loans receivable, PCI	10,259	11,416
Total loans receivable, Loans individually evaluated for impairment	0	4
Total loans receivable, Loans collectively evaluated	10,259	11,412
Total loans	10,259	11,416
Retail Consumer Loans | Consumer | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	0
Commercial Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	20,984	14,809	$ 13,050	$ 11,839
Total loans	1,861,588	1,542,722
Commercial Loans | Commercial real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	961	28
Allowance for loan losses, Loans collectively evaluated	10,731	7,890
Allowance for loan losses, Total	11,805	8,036
Total loans receivable, PCI	1,047,084	919,586
Total loans receivable, Loans individually evaluated for impairment	7,924	8,692
Total loans receivable, Loans collectively evaluated	1,039,160	910,863
Total loans	1,052,906	927,261
Commercial Loans | Commercial real estate | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	113	118
Total loans receivable, PCI	5,822	7,706
Commercial Loans | Construction and development
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	5	5
Allowance for loan losses, Loans collectively evaluated	3,599	3,187
Allowance for loan losses, Total	3,608	3,196
Total loans receivable, PCI	215,344	209,975
Total loans receivable, Loans individually evaluated for impairment	299	1,397
Total loans receivable, Loans collectively evaluated	215,045	208,578
Total loans	215,934	210,916
Commercial Loans | Construction and development | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	4	4
Total loans receivable, PCI	590	941
Commercial Loans | Commercial and industrial
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	31	2
Allowance for loan losses, Loans collectively evaluated	2,153	1,957
Allowance for loan losses, Total	2,199	1,976
Total loans receivable, PCI	153,263	158,802
Total loans receivable, Loans individually evaluated for impairment	852	2
Total loans receivable, Loans collectively evaluated	152,411	158,800
Total loans	154,825	160,471
Commercial Loans | Commercial and industrial | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	15	17
Total loans receivable, PCI	1,562	1,669
Commercial Loans | Equipment finance
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	209	0
Allowance for loan losses, Loans collectively evaluated	2,598	1,305
Allowance for loan losses, Total	2,807	1,305
Total loans receivable, PCI	229,239	132,058
Total loans receivable, Loans individually evaluated for impairment	801	0
Total loans receivable, Loans collectively evaluated	228,438	132,058
Total loans	229,239	132,058
Commercial Loans | Equipment finance | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	0
Commercial Loans | Municipal leases
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0	0
Allowance for loan losses, Loans collectively evaluated	697	435
Allowance for loan losses, Total	697	435
Total loans receivable, PCI	127,987	112,016
Total loans receivable, Loans individually evaluated for impairment	0	0
Total loans receivable, Loans collectively evaluated	127,987	112,016
Total loans	127,987	112,016
Commercial Loans | Municipal leases | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0	0
Total loans receivable, PCI	0	0
Commercial Loans | paycheck protection program [Domain]
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, Loans individually evaluated for impairment	0
Allowance for loan losses, Loans collectively evaluated	0
Allowance for loan losses, Total	0
Total loans receivable, PCI	80,697
Total loans receivable, Loans individually evaluated for impairment	0
Total loans receivable, Loans collectively evaluated	80,697
Total loans	80,697	$ 0
Commercial Loans | paycheck protection program [Domain] | Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Allowance for loan losses, PCI	0
Total loans receivable, PCI	$ 0