Loans, Loans Held For Sale, Notes and Certificates and Loan Servicing Rights - Past Due Table (Details) $ in Thousands	Sep. 30, 2017 USD ($) Loan	Dec. 31, 2016 USD ($) Loan
Loans and Loans Held for Sale
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Outstanding principal balance, past 90 days due	$ 42,258	$ 45,718
Outstanding principal balance, non-accrual loans	4,945	5,055
Net fair value adjustments, past 90 days due	(35,499)	(40,183)
Net fair value adjustments, non-accrual loans	(4,123)	(4,392)
Fair value, past 90 days due	6,759	5,535
Fair value, non-accrual loans	$ 822	$ 663
Number of loans, 90 days past due | Loan	4,023	4,041
Number of loans, non-accrual loans | Loan	507	483
Loans Invested in by Company
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Outstanding principal balance, past 90 days due	$ 1,087	$ 511
Outstanding principal balance, non-accrual loans	189	90
Net fair value adjustments, past 90 days due	(932)	(449)
Net fair value adjustments, non-accrual loans	(165)	(80)
Fair value, past 90 days due	155	62
Fair value, non-accrual loans	$ 24	$ 10
Number of loans, 90 days past due | Loan	265	154
Number of loans, non-accrual loans | Loan	38	18