Loans Held for Investment - Schedule of Loans Held for Investment Contractually Repricing (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Jun. 30, 2020
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	$ 34,144
Total loans held for investment, gross	852,082	$ 904,466
Mortgage loans, Single-family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	32,222
Total loans held for investment, gross	268,272	298,810
Mortgage Loans, Multi Family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	219
Total loans held for investment, gross	484,408	491,903
Mortgage loans, Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	153
Total loans held for investment, gross	95,279	105,235
Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	1,062
Total loans held for investment, gross	3,040	7,801
Mortgage Loans Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	139
Total loans held for investment, gross	139	143
Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	349
Total loans held for investment, gross	849	480
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Fixed Rate	0
Total loans held for investment, gross	95	$ 94
Within One Year [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	267,550
Within One Year [Member] | Mortgage loans, Single-family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	61,331
Within One Year [Member] | Mortgage Loans, Multi Family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	160,784
Within One Year [Member] | Mortgage loans, Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	42,862
Within One Year [Member] | Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	1,978
Within One Year [Member] | Mortgage Loans Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
Within One Year [Member] | Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	500
Within One Year [Member] | Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	95
After One Year Through 3 Years [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	202,545
After One Year Through 3 Years [Member] | Mortgage loans, Single-family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	41,705
After One Year Through 3 Years [Member] | Mortgage Loans, Multi Family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	127,637
After One Year Through 3 Years [Member] | Mortgage loans, Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	33,203
After One Year Through 3 Years [Member] | Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After One Year Through 3 Years [Member] | Mortgage Loans Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After One Year Through 3 Years [Member] | Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After One Year Through 3 Years [Member] | Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 3 Years Through 5 Years [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	229,807
After 3 Years Through 5 Years [Member] | Mortgage loans, Single-family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	42,063
After 3 Years Through 5 Years [Member] | Mortgage Loans, Multi Family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	168,683
After 3 Years Through 5 Years [Member] | Mortgage loans, Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	19,061
After 3 Years Through 5 Years [Member] | Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 3 Years Through 5 Years [Member] | Mortgage Loans Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 3 Years Through 5 Years [Member] | Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 3 Years Through 5 Years [Member] | Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 5 Years Through 10 Years [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	118,036
After 5 Years Through 10 Years [Member] | Mortgage loans, Single-family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	90,951
After 5 Years Through 10 Years [Member] | Mortgage Loans, Multi Family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	27,085
After 5 Years Through 10 Years [Member] | Mortgage loans, Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 5 Years Through 10 Years [Member] | Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 5 Years Through 10 Years [Member] | Mortgage Loans Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 5 Years Through 10 Years [Member] | Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	0
After 5 Years Through 10 Years [Member] | Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable, Adjustable Rate	$ 0