Loans - Schedule of Allowance for Credit Losses on Financing Receivables Table (Details) - USD ($) $ in Thousands	12 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2018
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Balance at beginning of period	$ 21,429	$ 21,060	$ 21,151
Provision for (recovery of) loan losses	8,500	5,700	0
Charge-offs	(3,816)	(7,409)	(2,647)
Recoveries	1,959	2,078	2,556
Balance at end of period	28,072	21,429	21,060
Retail Consumer Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Balance at beginning of period	6,419	7,527	8,585
Provision for (recovery of) loan losses	(137)	(1,244)	(906)
Charge-offs	(855)	(1,136)	(1,142)
Recoveries	1,479	1,272	990
Balance at end of period	6,906	6,419	7,527
Commercial Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Balance at beginning of period	14,809	13,050	11,839
Provision for (recovery of) loan losses	8,656	7,226	678
Charge-offs	(2,961)	(6,273)	(1,033)
Recoveries	480	806	1,566
Balance at end of period	20,984	14,809	13,050
Purchased Credit Impaired (PCI) Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Balance at beginning of period	201	483	727
Provision for (recovery of) loan losses	(19)	(282)	228
Charge-offs	0	0	(472)
Recoveries	0	0	0
Balance at end of period	$ 182	$ 201	$ 483