Loans - Schedule of PCI Loans Credit Quality Indicators (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Jun. 30, 2019
Financing Receivable, Recorded Investment
Total loans	$ 2,756,138	$ 2,688,436
Pass
Financing Receivable, Recorded Investment
Total loans	2,694,017	2,650,990
Special Mention
Financing Receivable, Recorded Investment
Total loans	33,803	12,968
Substandard
Financing Receivable, Recorded Investment
Total loans	28,095	24,058
Doubtful
Financing Receivable, Recorded Investment
Total loans	207	388
Loss
Financing Receivable, Recorded Investment
Total loans	16	32
Retail Consumer Loans | One-to-four family
Financing Receivable, Recorded Investment
Total loans	469,220	654,723
Retail Consumer Loans | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	137,447	139,210
Retail Consumer Loans | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	81,514	80,230
Retail Consumer Loans | Pass | One-to-four family
Financing Receivable, Recorded Investment
Total loans	458,248	644,159
Retail Consumer Loans | Pass | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	134,697	137,001
Retail Consumer Loans | Pass | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	81,112	79,995
Retail Consumer Loans | Special Mention | One-to-four family
Financing Receivable, Recorded Investment
Total loans	1,724	2,089
Retail Consumer Loans | Special Mention | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	902	766
Retail Consumer Loans | Special Mention | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	71
Retail Consumer Loans | Substandard | One-to-four family
Financing Receivable, Recorded Investment
Total loans	9,042	8,072
Retail Consumer Loans | Substandard | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	1,848	1,434
Retail Consumer Loans | Substandard | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	402	164
Retail Consumer Loans | Doubtful | One-to-four family
Financing Receivable, Recorded Investment
Total loans	206	384
Retail Consumer Loans | Doubtful | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	0	0
Retail Consumer Loans | Doubtful | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	0
Retail Consumer Loans | Loss | One-to-four family
Financing Receivable, Recorded Investment
Total loans	0	19
Retail Consumer Loans | Loss | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	0	9
Retail Consumer Loans | Loss | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	0
Commercial Loans | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	1,047,084	919,586
Commercial Loans | Construction and development
Financing Receivable, Recorded Investment
Total loans	215,344	209,975
Commercial Loans | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	153,263	158,802
Commercial Loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	1,028,709	901,214
Commercial Loans | Pass | Construction and development
Financing Receivable, Recorded Investment
Total loans	212,370	207,827
Commercial Loans | Pass | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	130,202	157,325
Commercial Loans | Special Mention | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	7,580	8,066
Commercial Loans | Special Mention | Construction and development
Financing Receivable, Recorded Investment
Total loans	2,723	790
Commercial Loans | Special Mention | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	20,439	877
Commercial Loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	10,779	10,306
Commercial Loans | Substandard | Construction and development
Financing Receivable, Recorded Investment
Total loans	250	1,357
Commercial Loans | Substandard | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	2,622	600
Commercial Loans | Doubtful | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	0	0
Commercial Loans | Doubtful | Construction and development
Financing Receivable, Recorded Investment
Total loans	1	1
Commercial Loans | Doubtful | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	0	0
Commercial Loans | Loss | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	16	0
Commercial Loans | Loss | Construction and development
Financing Receivable, Recorded Investment
Total loans	0	0
Commercial Loans | Loss | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans
Financing Receivable, Recorded Investment
Total loans	12,792	16,781
Purchased Credit Impaired (PCI) Loans | Pass
Financing Receivable, Recorded Investment
Total loans	8,110	11,113
Purchased Credit Impaired (PCI) Loans | Special Mention
Financing Receivable, Recorded Investment
Total loans	2,207	2,151
Purchased Credit Impaired (PCI) Loans | Substandard
Financing Receivable, Recorded Investment
Total loans	2,472	3,514
Purchased Credit Impaired (PCI) Loans | Doubtful
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Loss
Financing Receivable, Recorded Investment
Total loans	3	3
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | One-to-four family
Financing Receivable, Recorded Investment
Total loans	4,473	5,868
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans	0	225
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	345	372
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Pass | One-to-four family
Financing Receivable, Recorded Investment
Total loans	2,994	4,124
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Pass | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans		225
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Pass | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	108	142
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Special Mention | One-to-four family
Financing Receivable, Recorded Investment
Total loans	465	248
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Special Mention | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans		0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Special Mention | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Substandard | One-to-four family
Financing Receivable, Recorded Investment
Total loans	1,014	1,496
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Substandard | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans		0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Substandard | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	237	230
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Doubtful | One-to-four family
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Doubtful | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans		0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Doubtful | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Loss | One-to-four family
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Loss | HELOCs - originated
Financing Receivable, Recorded Investment
Total loans		0
Purchased Credit Impaired (PCI) Loans | Retail Consumer Loans | Loss | Construction and land/lots
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	5,822	7,706
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Construction and development
Financing Receivable, Recorded Investment
Total loans	590	941
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	1,562	1,669
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	3,181	4,503
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Pass | Construction and development
Financing Receivable, Recorded Investment
Total loans	271	453
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Pass | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	1,556	1,666
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Special Mention | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	1,742	1,903
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Special Mention | Construction and development
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Special Mention | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	899	1,300
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Substandard | Construction and development
Financing Receivable, Recorded Investment
Total loans	319	488
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Substandard | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	3	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Doubtful | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Doubtful | Construction and development
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Doubtful | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Loss | Commercial real estate
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Loss | Construction and development
Financing Receivable, Recorded Investment
Total loans	0	0
Purchased Credit Impaired (PCI) Loans | Commercial Loans | Loss | Commercial and industrial
Financing Receivable, Recorded Investment
Total loans	$ 3	$ 3