Fair Value of Financial Instruments - Schedule of Reconciliation of Beginning and Ending Balances of Recurring Fair Value Measurements Using Level 3 Inputs (Details) - Fair Value, Inputs, Level 3 - USD ($) $ in Thousands	12 Months Ended
	Jun. 30, 2021	Jun. 30, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	$ 2,469	$ 5,379
Total gains or losses (realized/unrealized) Included in earnings	51	13
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	5	(16)
Purchases	0	0
Issuances	0	0
Settlements	(487)	(2,907)
Transfers in and/or out of Level 3	0	0
Ending balance	2,038	2,469
Private issue CMO
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	197	269
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	9	(14)
Purchases	0	0
Issuances	0	0
Settlements	(52)	(58)
Transfers in and/or out of Level 3	0	0
Ending balance	154	197
Loans Held For Investment, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	2,258	5,094
Total gains or losses (realized/unrealized) Included in earnings	51	13
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	0	0
Purchases	0	0
Issuances	0	0
Settlements	(435)	(2,849)
Transfers in and/or out of Level 3	0	0
Ending balance	1,874	2,258
Unrealized gains and losses on Interest-Only Strips
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	14	16
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	(4)	(2)
Purchases	0	0
Issuances	0	0
Settlements	0	0
Transfers in and/or out of Level 3	0	0
Ending balance	$ 10	$ 14