Fair Value of Financial Instruments - Assets and Liabilities Measured at Fair Value on Non-Recurring Basis (Details) - Fair Value, Measurements, Nonrecurring - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Guarantee obligation		$ 4,414
Gain (Loss) on assets measured at fair value on a non-recurring basis		0
Level 1
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Guarantee obligation		0
Level 2
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Guarantee obligation		0
Level 3
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Guarantee obligation		4,414
Residential loans, at lower of cost or fair value
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	$ 867	1,096
Gain (Loss) on assets measured at fair value on a non-recurring basis	(17)	3
Residential loans, at lower of cost or fair value | Level 1
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	0	0
Residential loans, at lower of cost or fair value | Level 2
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	0	0
Residential loans, at lower of cost or fair value | Level 3
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	867	1,096
Commercial loans, at lower of cost or fair value
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	2,700
Gain (Loss) on assets measured at fair value on a non-recurring basis	(300)
Commercial loans, at lower of cost or fair value | Level 1
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	0
Commercial loans, at lower of cost or fair value | Level 2
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	0
Commercial loans, at lower of cost or fair value | Level 3
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
Loans, at lower of cost or fair value	2,700
REO
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
REO	5,207	2,395
Gain (Loss) on assets measured at fair value on a non-recurring basis	(1,831)	(764)
REO | Level 1
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
REO	0	0
REO | Level 2
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
REO	0	0
REO | Level 3
Fair Value, Assets and Liabilities Measured on Nonrecurring Basis [Line Items]
REO	$ 5,207	$ 2,395