Loans Held for Investment - Schedule of Gross Loans Held for Investment by Loan Types and Risk Category (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Jun. 30, 2020
Loans and Leases Receivable, Gross	$ 886,849	$ 904,466
Mortgage loans, Single-family
Loans and Leases Receivable, Gross	288,790	298,810
Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	482,900	491,903
Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	105,207	105,235
Construction
Loans and Leases Receivable, Gross	8,787	7,801
Mortgage Loans Other
Loans and Leases Receivable, Gross	142	143
Commercial business loans
Loans and Leases Receivable, Gross	923	480
Consumer loans
Loans and Leases Receivable, Gross	100	94
Pass
Loans and Leases Receivable, Gross	875,866	890,083
Pass | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	281,593	289,942
Pass | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	479,145	488,126
Pass | Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	105,207	105,235
Pass | Construction
Loans and Leases Receivable, Gross	8,787	6,098
Pass | Mortgage Loans Other
Loans and Leases Receivable, Gross	142	143
Pass | Commercial business loans
Loans and Leases Receivable, Gross	892	445
Pass | Consumer loans
Loans and Leases Receivable, Gross	100	94
Special Mention
Loans and Leases Receivable, Gross	5,930	8,600
Special Mention | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	2,175	3,120
Special Mention | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	3,755	3,777
Special Mention | Construction
Loans and Leases Receivable, Gross		1,703
Substandard
Loans and Leases Receivable, Gross	5,053	5,783
Substandard | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	5,022	5,748
Substandard | Commercial business loans
Loans and Leases Receivable, Gross	$ 31	$ 35