Parent Company Financial Information - Condensed Income Statement (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2018
Income:
Interest income	$ 31,074	$ 33,037	$ 35,896	$ 36,247	$ 35,820	$ 34,714	$ 34,400	$ 32,280	$ 136,254	$ 137,214	$ 117,402
Expense:
Loss on sale and impairment of REO									536	439	127
Recovery of loan losses	2,700	5,400	400	0	200	5,500	0	0
Income Tax Expense	964	188	2,476	2,396	2,107	185	2,287	2,212	6,024	6,791	26,736
Net Income	$ 3,595	$ 1,193	$ 9,191	$ 8,804	$ 8,013	$ 3,302	$ 8,041	$ 7,790	22,783	27,146	8,235
Parent Company
Income:
Interest income									217	329	456
Other income									1	54	44
Equity earnings in Bank subsidiary									23,522	27,287	8,427
Total income									23,740	27,670	8,927
Expense:
Management fee expense									399	407	385
REO expense									5	11	34
Loss on sale and impairment of REO									249	114	158
Recovery of loan losses									(4)	(259)	(131)
Other expense									258	251	246
Total expense									907	524	692
Income Before Income Taxes									22,833	27,146	8,235
Income Tax Expense									50	0	0
Net Income									$ 22,783	$ 27,146	$ 8,235