Loans Held for Investment - Schedule of Allowance For Loan Losses and Recorded Investment in Gross Loans, by Portfolio Type (Details) - USD ($) $ in Thousands	3 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Jun. 30, 2020
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 8,265	$ 7,076
Provision (recovery) for loan losses	220	(181)
Recoveries	5	36
Charge-offs	0	(2)
Net (charge-offs) recoveries	5	34
Balance, end of the period	8,490	6,929
Total individually evaluated allowance	84	54
Total collectively evaluated allowance	8,406	6,875
Individually evaluated for impairment	2,988	4,950
Collectively evaluated for impairment	883,861	920,055
Total Loans Held for Investment, Gross	$ 886,849	$ 925,005	$ 904,466
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	0.95%	0.74%
Net (recoveries) charge-offs as a percentage of average loans receivable, net, during the period (annualized)	0.00%	(0.02%)
Mortgage loans, Single-family
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 2,622	$ 2,709
Provision (recovery) for loan losses	44	(510)
Recoveries	5	36
Charge-offs	0	(1)
Balance, end of the period	2,671	2,234
Total individually evaluated allowance	80	47
Total collectively evaluated allowance	2,591	2,187
Individually evaluated for impairment	2,957	3,766
Collectively evaluated for impairment	285,833	324,566
Total Loans Held for Investment, Gross	$ 288,790	$ 328,332	298,810
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	0.92%	0.68%
Mortgage Loans, Multi Family
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 4,329	$ 3,219
Provision (recovery) for loan losses	161	288
Recoveries	0
Charge-offs	0
Balance, end of the period	4,490	3,507
Total collectively evaluated allowance	4,490	3,507
Collectively evaluated for impairment	482,900	479,597
Total Loans Held for Investment, Gross	$ 482,900	$ 479,597	491,903
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	0.93%	0.73%
Mortgage loans, Commercial real estate
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 1,110	$ 1,050
Provision (recovery) for loan losses	52	35
Recoveries	0
Charge-offs	0
Balance, end of the period	1,162	1,085
Total collectively evaluated allowance	1,162	1,085
Collectively evaluated for impairment	105,207	110,652
Total Loans Held for Investment, Gross	$ 105,207	$ 110,652	105,235
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	1.10%	0.98%
Construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 171	$ 61
Provision (recovery) for loan losses	(55)	13
Recoveries	0
Charge-offs	0
Balance, end of the period	116	74
Total collectively evaluated allowance	116	74
Individually evaluated for impairment		1,139
Collectively evaluated for impairment	8,787	4,773
Total Loans Held for Investment, Gross	$ 8,787	$ 5,912	7,801
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	1.32%	1.25%
Mortgage Loans Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 3	$ 3
Provision (recovery) for loan losses	0	(3)
Recoveries	0
Charge-offs	0
Balance, end of the period	3
Total collectively evaluated allowance	3
Collectively evaluated for impairment	142
Total Loans Held for Investment, Gross	$ 142		143
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	2.11%
Commercial business loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 24	26
Provision (recovery) for loan losses	18	(6)
Recoveries	0
Charge-offs	0
Balance, end of the period	42	20
Total individually evaluated allowance	4	7
Total collectively evaluated allowance	38	13
Individually evaluated for impairment	31	45
Collectively evaluated for impairment	892	323
Total Loans Held for Investment, Gross	$ 923	$ 368	480
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	4.55%	5.43%
Consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 6	$ 8
Provision (recovery) for loan losses	0	2
Recoveries	0
Charge-offs	0	(1)
Balance, end of the period	6	9
Total collectively evaluated allowance	6	9
Collectively evaluated for impairment	100	144
Total Loans Held for Investment, Gross	$ 100	$ 144	$ 94
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	6.00%	6.25%