Loans - Schedule of Accounts, Notes, Loans and Financing Receivable (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Jun. 30, 2019
Accounts, Notes, Loans and Financing Receivable
Total loans	$ 2,768,930	$ 2,705,186
Deferred loan costs, net	189	4
Total loans, net of deferred loan fees and discount	2,769,119	2,705,190
Allowance for loan losses	(28,072)	(21,429)
Net loans	2,741,047	2,683,761
Retail Consumer Loans
Accounts, Notes, Loans and Financing Receivable
Total loans	907,342	1,162,464
Retail Consumer Loans | One-to-four family
Accounts, Notes, Loans and Financing Receivable
Total loans	473,693	660,591
Retail Consumer Loans | HELOCs - originated
Accounts, Notes, Loans and Financing Receivable
Total loans	137,447	139,435
Retail Consumer Loans | HELOCs - purchased
Accounts, Notes, Loans and Financing Receivable
Total loans	71,781	116,972
Retail Consumer Loans | Construction and land/lots
Accounts, Notes, Loans and Financing Receivable
Total loans	81,859	80,602
Retail Consumer Loans | Indirect auto finance
Accounts, Notes, Loans and Financing Receivable
Total loans	132,303	153,448
Retail Consumer Loans | Consumer
Accounts, Notes, Loans and Financing Receivable
Total loans	10,259	11,416
Commercial Loans
Accounts, Notes, Loans and Financing Receivable
Total loans	1,861,588	1,542,722
Commercial Loans | Commercial real estate
Accounts, Notes, Loans and Financing Receivable
Total loans	1,052,906	927,261
Commercial Loans | Construction and development
Accounts, Notes, Loans and Financing Receivable
Total loans	215,934	210,916
Commercial Loans | Commercial and industrial
Accounts, Notes, Loans and Financing Receivable
Total loans	154,825	160,471
Commercial Loans | Equipment finance
Accounts, Notes, Loans and Financing Receivable
Total loans	229,239	132,058
Commercial Loans | Municipal leases
Accounts, Notes, Loans and Financing Receivable
Total loans	127,987	112,016
Commercial Loans | paycheck protection program [Domain]
Accounts, Notes, Loans and Financing Receivable
Total loans	80,697	0
Executive Officers and Directors
Accounts, Notes, Loans and Financing Receivable
Net loans	1,498	1,800
Executive Officers and Directors | Unfunded Loan Commitment
Accounts, Notes, Loans and Financing Receivable
Net loans	$ 54	$ 118