Fair Value of Financial Instruments - Schedule of Reconciliation of Beginning and Ending Balances of Recurring Fair Value Measurements Using Level 3 Inputs (Details) - Fair Value, Inputs, Level 3 - USD ($) $ in Thousands	3 Months Ended
	Sep. 30, 2021	Sep. 30, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	$ 2,038	$ 2,469
Total gains or losses (realized/unrealized) Included in earnings	2	(4)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	0	3
Purchases	0	0
Issuances	0	0
Settlements	(304)	(31)
Transfers in and/or out of Level 3	0	0
Ending balance	1,736	2,437
Private issue CMO
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	154	197
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	1	4
Purchases	0	0
Issuances	0	0
Settlements	(5)	(17)
Transfers in and/or out of Level 3	0	0
Ending balance	150	184
Loans Held For Investment at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	1,874	2,258
Total gains or losses (realized/unrealized) Included in earnings	2	(4)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	0	0
Purchases	0	0
Issuances	0	0
Settlements	(299)	(14)
Transfers in and/or out of Level 3	0	0
Ending balance	1,577	2,240
Interest-Only Strips
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	10	14
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	(1)	(1)
Purchases	0	0
Issuances	0	0
Settlements	0	0
Transfers in and/or out of Level 3	0	0
Ending balance	$ 9	$ 13