Fair Value of Financial Instruments - Schedule of Reconciliation of Beginning and Ending Balances of Recurring Fair Value Measurements Using Level 3 Inputs (Details) - Fair Value, Inputs, Level 3 - USD ($) $ in Thousands	3 Months Ended
	Sep. 30, 2020	Sep. 30, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	$ 2,469	$ 5,379
Total gains or losses (realized/unrealized) Included in earnings	(4)	(18)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	(3)	(2)
Purchases	0	0
Issuances	0	0
Settlements	(31)	(706)
Transfers in and/or out of Level 3	0	0
Ending balance	2,437	4,653
Private issue CMO
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	197	269
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	4	0
Purchases	0	0
Issuances	0	0
Settlements	(17)	(16)
Transfers in and/or out of Level 3	0	0
Ending balance	184	253
Loans Held For Investment at Fair Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	2,258	5,094
Total gains or losses (realized/unrealized) Included in earnings	(4)	(18)
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	0	0
Purchases	0	0
Issuances	0	0
Settlements	(14)	(690)
Transfers in and/or out of Level 3	0	0
Ending balance	2,240	4,386
Interest-Only Strips
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Beginning balance	14	16
Total gains or losses (realized/unrealized) Included in earnings	0	0
Total gains or losses (realized/unrealized) Included in other comprehensive income (loss)	(1)	(2)
Purchases	0	0
Issuances	0	0
Settlements	0	0
Transfers in and/or out of Level 3	0	0
Ending balance	$ 13	$ 14