Fair Value of Financial Instruments - Valuation techniques and inputs used (Details) $ in Thousands	3 Months Ended
	Sep. 30, 2021 USD ($)
Private issue CMO | Weighted Average | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	2.30%
Loans Held For Investment at Fair Value | Weighted Average | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	101.00%
Loans Held For Investment at Fair Value | Weighted Average | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	4.60%
Non-performing loans one | Weighted Average | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	28.40%
Mortgage servicing assets | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	26.50%
Mortgage servicing assets | Weighted Average | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.10%
Interest-Only Strips | Weighted Average | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	28.70%
Fair Value, Inputs, Level 3 | Private issue CMO
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 150
Fair Value, Inputs, Level 3 | Private issue CMO | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Increase
Fair Value, Inputs, Level 3 | Private issue CMO | Minimum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	1.60%
Fair Value, Inputs, Level 3 | Private issue CMO | Maximum | Market comparable pricing | Comparability adjustment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	2.50%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 1,577
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Increase
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Minimum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	98.00%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Minimum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	1.30%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Maximum | Relative value analysis | Broker quotes
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	104.10%
Fair Value, Inputs, Level 3 | Loans Held For Investment at Fair Value | Maximum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	100.00%
Fair Value, Inputs, Level 3 | Non-performing loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 4,931
Fair Value, Inputs, Level 3 | Non-performing loans | Discounted cash flow | Default rates.
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Non-performing loans | Minimum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	5.00%
Fair Value, Inputs, Level 3 | Non-performing loans | Maximum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	5.20%
Fair Value, Inputs, Level 3 | Non-performing loans | Weighted Average | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	5.00%
Fair Value, Inputs, Level 3 | Non-performing loans one
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 972
Fair Value, Inputs, Level 3 | Non-performing loans one | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Non-performing loans one | Minimum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	20.00%
Fair Value, Inputs, Level 3 | Non-performing loans one | Maximum | Relative value analysis | Credit risk factors
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	30.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 132
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	21.10%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Minimum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	9.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	60.00%
Fair Value, Inputs, Level 3 | Mortgage servicing assets | Maximum | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	10.50%
Fair Value, Inputs, Level 3 | Interest-Only Strips
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, Fair Value Disclosure	$ 9
Fair Value, Inputs, Level 3 | Interest-Only Strips | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Fair Value, Inputs, Level 3 | Interest-Only Strips | Discounted cash flow | Discount rate
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Impact to valuation from an increase in inputs on assets	Decrease
Assets Fair Value Measurement Input	9.00%
Fair Value, Inputs, Level 3 | Interest-Only Strips | Minimum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	25.40%
Fair Value, Inputs, Level 3 | Interest-Only Strips | Maximum | Discounted cash flow | Prepayment speed (CPR)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets Fair Value Measurement Input	28.90%