Income Taxes - Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Jun. 30, 2020
Deferred Tax Assets [Abstract]
Loss reserves	$ 2,734	$ 3,034
Non-accrued interest	392	326
Deferred compensation	2,512	2,824
Accrued vacation	197	177
Depreciation	153	90
Other	307	395
Total deferred tax assets	6,295	6,846
Deferred Tax Liabilities [Abstract]
FHLB - San Francisco stock dividends	(645)	(645)
Prepaid expenses	(41)	(41)
Unrealized gain on investment securities	(27)	(40)
Unrealized gain on interest-only strips	(3)	(4)
Deferred loan costs	(2,974)	(3,071)
State tax	(62)	(34)
Total deferred tax liabilities	(3,752)	(3,835)
Net deferred tax assets	$ 2,543	$ 3,011