Investment Securities - Schedule of amortized cost and estimated fair value of Held to maturity investments (Details) - USD ($) $ in Thousands	Sep. 30, 2020		Jun. 30, 2020
Held to maturity
Amortized Cost	$ 193,868		$ 118,627
Gross Unrealized Gains	2,844		2,636
Gross Unrealized (Losses)	(165)		(62)
Estimated Fair Value	196,547		121,201
Carrying Value	193,868		118,627
U.S. government sponsored enterprise MBS
Held to maturity
Amortized Cost	191,224	[1]	115,763
Gross Unrealized Gains	2,844	[1]	2,636
Gross Unrealized (Losses)	(147)	[1]	(45)
Estimated Fair Value	193,921	[1]	118,354
Carrying Value	191,224	[1]	115,763
U.S. SBA securities
Held to maturity
Amortized Cost	2,044	[2]	2,064
Gross Unrealized Gains			0
Gross Unrealized (Losses)	(18)	[2]	(17)
Estimated Fair Value	2,026	[2]	2,047
Carrying Value	2,044	[2]	2,064
Certificate of deposits
Held to maturity
Amortized Cost	600		800
Gross Unrealized Gains	0		0
Gross Unrealized (Losses)	0		0
Estimated Fair Value	600		800
Carrying Value	$ 600		$ 800

[1] Mortgage-Backed Securities ("MBS").

[2] Small Business Administration ("SBA").