Loans Held for Investment - Schedule of Gross Loans Held for Investment by Loan Types and Risk Category (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Jun. 30, 2020
Loans and Leases Receivable, Gross	$ 857,080	$ 904,466
Mortgage loans, Single-family
Loans and Leases Receivable, Gross	257,864	298,810
Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	488,412	491,903
Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	102,551	105,235
Construction
Loans and Leases Receivable, Gross	7,135	7,801
Mortgage Loans Other
Loans and Leases Receivable, Gross	141	143
Commercial business loans
Loans and Leases Receivable, Gross	882	480
Consumer loans
Loans and Leases Receivable, Gross	95	94
Pass
Loans and Leases Receivable, Gross	841,231	890,083
Pass | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	245,747	289,942
Pass | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	484,680	488,126
Pass | Mortgage loans, Commercial real estate
Loans and Leases Receivable, Gross	102,551	105,235
Pass | Construction
Loans and Leases Receivable, Gross	7,135	6,098
Pass | Mortgage Loans Other
Loans and Leases Receivable, Gross	141	143
Pass | Commercial business loans
Loans and Leases Receivable, Gross	882	445
Pass | Consumer loans
Loans and Leases Receivable, Gross	95	94
Special Mention
Loans and Leases Receivable, Gross	4,667	8,600
Special Mention | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	935	3,120
Special Mention | Mortgage Loans, Multi Family
Loans and Leases Receivable, Gross	3,732	3,777
Special Mention | Construction
Loans and Leases Receivable, Gross		1,703
Substandard
Loans and Leases Receivable, Gross	11,182	5,783
Substandard | Mortgage loans, Single-family
Loans and Leases Receivable, Gross	$ 11,182	5,748
Substandard | Commercial business loans
Loans and Leases Receivable, Gross		$ 35