Loans Held for Investment - Schedule of Allowance For Loan Losses and Recorded Investment in Gross Loans, by Portfolio Type (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2020	Dec. 31, 2019	Jun. 30, 2020
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 8,490	$ 6,929	$ 8,265	$ 7,076
Provision (recovery) for loan losses	39	(22)	259	(203)
Recoveries	10	14	15	50
Charge-offs	(1)	0	(1)	(2)
Net recoveries (charge-offs)	9	14	14	48
Balance, end of the period	8,538	6,921	8,538	6,921
Total individually evaluated allowance	570	52	570	52
Total collectively evaluated allowance	7,968	6,869	7,968	6,869
Individually evaluated for impairment	9,237	3,096	9,237	3,096
Collectively evaluated for impairment	847,843	938,644	847,843	938,644
Total Loans Held for Investment, Gross	$ 857,080	$ 941,740	$ 857,080	$ 941,740	$ 904,466
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	0.99%	0.73%	0.99%	0.73%
Net (recoveries) charge-offs as a percentage of average loans receivable, net, during the period (annualized)	0.00%	(0.01%)	0.00%	(0.01%)
Mortgage loans, Single-family
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 2,671	$ 2,234	$ 2,622	$ 2,709
Provision (recovery) for loan losses	26	(90)	70	(600)
Recoveries	9	13	14	49
Charge-offs	0	0	0	(1)
Balance, end of the period	2,706	2,157	2,706	2,157
Total individually evaluated allowance	570	46	570	46
Total collectively evaluated allowance	2,136	2,111	2,136	2,111
Individually evaluated for impairment	9,237	3,053	9,237	3,053
Collectively evaluated for impairment	248,627	344,291	248,627	344,291
Total Loans Held for Investment, Gross	$ 257,864	$ 347,344	$ 257,864	$ 347,344	298,810
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	1.05%	0.62%	1.05%	0.62%
Mortgage Loans, Multi Family
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 4,490	$ 3,507	$ 4,329	$ 3,219
Provision (recovery) for loan losses	50	(5)	211	283
Recoveries	0
Charge-offs	0
Balance, end of the period	4,540	3,502	4,540	3,502
Total collectively evaluated allowance	4,540	3,502	4,540	3,502
Collectively evaluated for impairment	488,412	479,151	488,412	479,151
Total Loans Held for Investment, Gross	$ 488,412	$ 479,151	$ 488,412	$ 479,151	491,903
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	0.93%	0.73%	0.93%	0.73%
Mortgage loans, Commercial real estate
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 1,162	$ 1,085	$ 1,110	$ 1,050
Provision (recovery) for loan losses	(30)	(27)	22	8
Recoveries	0
Charge-offs	0
Balance, end of the period	1,132	1,058	1,132	1,058
Total collectively evaluated allowance	1,132	1,058	1,132	1,058
Collectively evaluated for impairment	102,551	107,613	102,551	107,613
Total Loans Held for Investment, Gross	$ 102,551	$ 107,613	$ 102,551	$ 107,613	105,235
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	1.10%	0.98%	1.10%	0.98%
Construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 116	$ 74	$ 171	$ 61
Provision (recovery) for loan losses	(6)	94	(61)	107
Recoveries	0
Charge-offs	0
Balance, end of the period	110	168	110	168
Total collectively evaluated allowance	110	168	110	168
Collectively evaluated for impairment	7,135	6,914	7,135	6,914
Total Loans Held for Investment, Gross	$ 7,135	$ 6,914	$ 7,135	$ 6,914	7,801
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	1.54%	2.43%	1.54%	2.43%
Mortgage Loans Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 3	$ 20	$ 3	$ 3
Provision (recovery) for loan losses	0	8		(3)
Recoveries	0
Charge-offs	0
Balance, end of the period	3	28	3	28
Total individually evaluated allowance		6		6
Total collectively evaluated allowance	3	22	3	22
Individually evaluated for impairment		43		43
Collectively evaluated for impairment	141	535	141	535
Total Loans Held for Investment, Gross	$ 141	$ 578	$ 141	578	143
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	2.13%	4.84%	2.13%
Commercial business loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 42		$ 24	26
Provision (recovery) for loan losses	(1)		17	2
Recoveries	0
Charge-offs	0
Balance, end of the period	41	$ 28	41	28
Total individually evaluated allowance		6		6
Total collectively evaluated allowance	41	22	41	22
Individually evaluated for impairment		43		43
Collectively evaluated for impairment	882	535	882	535
Total Loans Held for Investment, Gross	$ 882	578	$ 882	$ 578	480
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	4.65%		4.65%	4.84%
Consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Balance, beginning of the period	$ 6	9	$ 6	$ 8
Provision (recovery) for loan losses	0	(2)
Recoveries	1	1	1	1
Charge-offs	(1)	0	(1)	(1)
Balance, end of the period	6	8	6	8
Total collectively evaluated allowance	6	8	6	8
Collectively evaluated for impairment	95	140	95	140
Total Loans Held for Investment, Gross	$ 95	$ 140	$ 95	$ 140	$ 94
Allowance for loan losses as a percentage of gross loans held for investment at the end of the period	6.32%	5.71%	6.32%	5.71%