Fair Value of Financial Instruments (Details 2) - USD ($) $ in Thousands		12 Months Ended
		Jun. 30, 2019	Jun. 30, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance		$ 6,400	$ 7,830
Total gains or losses (realized/unrealized) Included in earnings		(618)	(168)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)		(3)	(9)
Purchases		0	0
Issuances		0	0
Settlements		(1,360)	(2,344)
Transfers in and/or out of Level 3		960	1,091
Ending balance		5,379	6,400
Private issue CMO
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance		350	461
Total gains or losses (realized/unrealized) Included in earnings		0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)		4	(1)
Purchases		0	0
Issuances		0	0
Settlements		(85)	(110)
Transfers in and/or out of Level 3		0	0
Ending balance		269	350
Loans held-for investment, at-fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	[1]	5,234	6,445
Total gains or losses (realized/unrealized) Included in earnings	[1]	188	(60)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	[1]	0	0
Purchases	[1]	0	0
Issuances	[1]	0	0
Settlements	[1]	(1,288)	(2,242)
Transfers in and/or out of Level 3	[1]	960	1,091
Ending balance	[1]	5,094	5,234
Interest-Only Strips
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance		23	31
Total gains or losses (realized/unrealized) Included in earnings		0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)		(7)	(8)
Purchases		0	0
Issuances		0	0
Settlements		0	0
Transfers in and/or out of Level 3		0	0
Ending balance		16	23
Loan Commitments to Originate
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	[2]	825	809
Total gains or losses (realized/unrealized) Included in earnings	[2]	(825)	16
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	[2]	0	0
Purchases	[2]	0	0
Issuances	[2]	0	0
Settlements	[2]	0	0
Transfers in and/or out of Level 3	[2]	0	0
Ending balance	[2]	0	825
Mandatory Commitments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	[3]	(32)	47
Total gains or losses (realized/unrealized) Included in earnings	[3]	19	(87)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	[3]	0	0
Purchases	[3]	0	0
Issuances	[3]	0	0
Settlements	[3]	13	8
Transfers in and/or out of Level 3	[3]	0	0
Ending balance	[3]	0	(32)
Option contracts
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance		0	37
Total gains or losses (realized/unrealized) Included in earnings		0	(37)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)		0	0
Purchases		0	0
Issuances		0	0
Settlements		0	0
Transfers in and/or out of Level 3		0	0
Ending balance		$ 0	$ 0

[1] The valuation of loans held for investment at fair value includes management estimates of the specific credit risk attributes of each loan, in addition to the quoted secondary-market prices which account for interest rate characteristics.

[2] Consists of commitments to extend credit on loans to be held for sale.

[3] Consists of mandatory loan sale commitments.