Fair Value of Financial Instruments (Details 4) - Level 3 $ in Thousands	12 Months Ended
	Jun. 30, 2019 USD ($)
Private issue CMO
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 269
Private issue CMO | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Increase	[1]
Private issue CMO | Minimum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	2.50%	[2]
Private issue CMO | Maximum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	2.90%	[2]
Private issue CMO | Weighted Average | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	2.80%	[2]
Loans held-for investment, at-fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 5,094
Loans held-for investment, at-fair value | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Increase	[1]
Loans held-for investment, at-fair value | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Loans held-for investment, at-fair value | Minimum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	98.70%	[2]
Loans held-for investment, at-fair value | Minimum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	1.20%	[2]
Loans held-for investment, at-fair value | Maximum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	104.30%	[2]
Loans held-for investment, at-fair value | Maximum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	100.00%	[2]
Loans held-for investment, at-fair value | Weighted Average | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	102.00%	[2]
Loans held-for investment, at-fair value | Weighted Average | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	4.30%	[2]
Non-performing loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 693	[3]
Non-performing loans | Discounted cash flow | Default rates
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Assets Fair Value Measurement Input	5.00%	[2],[3]
Mortgage servicing assets
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 627
Mortgage servicing assets | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Mortgage servicing assets | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Mortgage servicing assets | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	14.60%	[2]
Mortgage servicing assets | Minimum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.00%	[2]
Mortgage servicing assets | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	60.00%	[2]
Mortgage servicing assets | Maximum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	10.50%	[2]
Mortgage servicing assets | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	23.90%	[2]
Mortgage servicing assets | Weighted Average | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.10%	[2]
Interest-only strips
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 16
Interest-only strips | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Interest-only strips | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Assets Fair Value Measurement Input	9.00%	[2]
Interest-only strips | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	19.70%	[2]
Interest-only strips | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	39.10%	[2]
Interest-only strips | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	37.70%	[2]
Non-performing loans one
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 1,554	[4]
Non-performing loans one | Relative value analysis | Credit risk factor
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease	[1]
Non-performing loans one | Minimum | Relative value analysis | Credit risk factor
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.00%	[2],[4]
Non-performing loans one | Maximum | Relative value analysis | Credit risk factor
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	30.00%	[2],[4]
Non-performing loans one | Weighted Average | Relative value analysis | Credit risk factor
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	19.90%	[2],[4]

[1] Unless otherwise noted, this column represents the directional change in the fair value of the Level 3 investments that would result from an increase to the corresponding unobservable input. A decrease to the unobservable input would have the opposite effect. Significant changes in these inputs in isolation could result in significantly higher or lower fair value measurements.

[2] The range is based on the historical estimated fair values and management estimates.

[3] Consist of restructured loans.

[4] Consist of other non-performing loans, excluding restructured loans.