Fair Value of Financial Instruments - Valuation techniques and inputs used (Details) $ in Thousands	6 Months Ended
	Dec. 31, 2020 USD ($)
Private issue CMO | Weighted Average | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	(0.30%)
Loans Held For Investment at Fair Value | Weighted Average | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	99.90%
Loans Held For Investment at Fair Value | Weighted Average | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	5.90%
Non-performing loans | Weighted Average | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.10%
Mortgage servicing assets | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	29.60%
Mortgage servicing assets | Weighted Average | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.10%
Interest-Only Strips | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	22.90%
Fair Value, Inputs, Level 3 | Private issue CMO
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 173
Fair Value, Inputs, Level 3 | Private issue CMO | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Increase
Fair Value, Inputs, Level 3 | Private issue CMO | Minimum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	(0.10%)
Fair Value, Inputs, Level 3 | Private issue CMO | Maximum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	(1.40%)
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 1,972
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Minimum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	97.60%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Minimum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	1.40%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Maximum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	102.00%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Maximum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	100.00%
Fair Value, Inputs, Level 3 | Non-performing loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 7,352
Fair Value, Inputs, Level 3 | Non-performing loans | Discounted cash flow | Default rates.
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Assets Fair Value Measurement Input	5.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 233
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.40%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Minimum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	60.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Maximum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	10.50%
Fair Value, Inputs, Level 3 | Interest-Only Strips
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 12
Fair Value, Inputs, Level 3 | Interest-Only Strips | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.00%
Fair Value, Inputs, Level 3 | Interest-Only Strips | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.40%
Fair Value, Inputs, Level 3 | Interest-Only Strips | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	23.00%
Fair Value, Inputs, Level 3 | Non-performing loans one
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 1,356
Fair Value, Inputs, Level 3 | Non-performing loans one | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Non-performing loans one | Minimum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.00%
Fair Value, Inputs, Level 3 | Non-performing loans one | Maximum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	30.00%