Loans Held for Investment - Schedule of restructured loan balances, net of allowance for loan losses, by loan type and by Nonaccrual Versus Accrual Status (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Jun. 30, 2020
Financing Receivable, Modifications [Line Items]
Restructured loans on non-accrual status	$ 6,983	$ 2,643
Restructured loans on accrual status	876
Restructured loans	7,859	2,643
Mortgage loans, Single-family
Financing Receivable, Modifications [Line Items]
Restructured loans on non-accrual status	6,983	2,612
Restructured loans on accrual status	876
Commercial business loans
Financing Receivable, Modifications [Line Items]
Restructured loans on non-accrual status	$ 0	$ 31