Income Taxes Income Taxes (Deferred Tax Assets and Liabilities) (Details) (USD $) In Thousands, unless otherwise specified	Jun. 30, 2013	Jun. 30, 2012
Deferred Tax Assets [Abstract]
Loss reserves	$ 9,341	$ 13,858
Non-accrued interest	420	237
Deferred compensation	3,106	3,425
Accrued vacation	323	281
Depreciation	0	28
State taxes	924	181
Other	202	159
Total deferred tax assets	14,316	18,169
Deferred Tax Liabilities [Abstract]
FHLB - San Francisco stock dividends	(2,069)	(3,015)
Unrealized gain on derivative financial instruments, at fair value	(2,916)	(1,238)
Unrealized gain on loans held for sale, at fair value	(1,422)	(2,133)
Unrealized gain on investment securities	(360)	(399)
Unrealized gain on interest-only strips	(41)	(53)
Deferred loan costs	(2,577)	(2,683)
Depreciation	(506)	0
Total deferred tax liabilities	(9,891)	(9,521)
Total net deferred tax assets	$ 4,425	$ 8,648