Loans Held for Investment - Schedule of Gross Loans Held for Investment by Loan Types and Risk Category (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Jun. 30, 2020
Loans and Leases Receivable, Gross	$ 852,082	$ 904,466
Mortgage loans, Single-family
Loans and Leases Receivable, Gross	268,272	298,810
Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	484,408	491,903
Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	95,279	105,235
Construction
Loans and Leases Receivable, Gross	3,040	7,801
Mortgage Loans Other
Loans and Leases Receivable, Gross	139	143
Commercial business loans
Loans and Leases Receivable, Gross	849	480
Consumer loans
Loans and Leases Receivable, Gross	95	94
Pass
Loans and Leases Receivable, Gross	840,908	890,083
Pass | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	258,217	289,942
Pass | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	483,289	488,126
Pass | Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	95,279	105,235
Pass | Construction
Loans and Leases Receivable, Gross	3,040	6,098
Pass | Mortgage Loans Other
Loans and Leases Receivable, Gross	139	143
Pass | Commercial business loans
Loans and Leases Receivable, Gross	849	445
Pass | Consumer loans
Loans and Leases Receivable, Gross	95	94
Special Mention
Loans and Leases Receivable, Gross	1,767	8,600
Special Mention | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	1,767	3,120
Special Mention | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	0	3,777
Special Mention | Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	0	0
Special Mention | Construction
Loans and Leases Receivable, Gross	0	1,703
Special Mention | Mortgage Loans Other
Loans and Leases Receivable, Gross	0
Special Mention | Commercial business loans
Loans and Leases Receivable, Gross	0	0
Special Mention | Consumer loans
Loans and Leases Receivable, Gross	0	0
Substandard
Loans and Leases Receivable, Gross	9,407	5,783
Substandard | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	8,288	5,748
Substandard | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	1,119	0
Substandard | Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	0	0
Substandard | Construction
Loans and Leases Receivable, Gross	0	0
Substandard | Mortgage Loans Other
Loans and Leases Receivable, Gross	0
Substandard | Commercial business loans
Loans and Leases Receivable, Gross	0	35
Substandard | Consumer loans
Loans and Leases Receivable, Gross	$ 0	$ 0