Quarterly Results of Operations (Unaudited) (Details) (USD $) In Thousands, except Per Share data, unless otherwise specified	3 Months Ended								12 Months Ended
	Jun. 30, 2013	Mar. 31, 2013	Dec. 31, 2012	Sep. 30, 2012	Jun. 30, 2012	Mar. 31, 2012	Dec. 31, 2011	Sep. 30, 2011	Jun. 30, 2013	Jun. 30, 2012	Jun. 30, 2011
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 10,085	$ 10,612	$ 11,617	$ 11,847	$ 12,518	$ 12,454	$ 13,452	$ 13,011	$ 44,161	$ 51,435
Interest expense	2,502	2,546	2,845	2,911	3,068	3,478	3,946	4,213	10,804	14,705	20,940
Net interest income	7,583	8,066	8,772	8,936	9,450	8,976	9,506	8,798	33,357	36,730	37,749
(Recovery) provision for loan losses	(1,538)	(517)	23	533	2,051	1,622	1,132	972	(1,499)	5,777	5,465
Net interest income, after (recovery) provision for loan losses	9,121	8,583	8,749	8,403	7,399	7,354	8,374	7,826	34,856	30,953
Non-interest income	17,618	15,388	20,035	22,159	15,980	11,309	7,313	8,548	75,200	43,150	36,357
Non-interest expense	17,519	15,729	16,769	17,326	15,991	14,597	12,474	12,303	67,343	55,365	45,372
Income before income tax benefit	9,220	8,242	12,015	13,236	7,388	4,066	3,213	4,071	42,713	18,738	23,269
Provision for income taxes	3,963	3,372	5,075	4,506	3,082	1,734	1,359	1,753	16,916	7,928	10,049
Net income	$ 5,257	$ 4,870	$ 6,940	$ 8,730	$ 4,306	$ 2,332	$ 1,854	$ 2,318	$ 25,797	$ 10,810	$ 13,220
Basic earnings per share (in dollars per share)	$ 0.51	$ 0.46	$ 0.65	$ 0.81	$ 0.39	$ 0.21	$ 0.16	$ 0.20	$ 2.43	$ 0.96	$ 1.16
Diluted earnings per share (in dollars per share)	$ 0.49	$ 0.45	$ 0.64	$ 0.80	$ 0.39	$ 0.21	$ 0.16	$ 0.20	$ 2.38	$ 0.96	$ 1.16