Fair Value of Financial Instruments - Assets Measured at Fair Value on a Nonrecurring Basis (Details) - Fair Value, Measurements, Nonrecurring - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015
Residential mortgage loans held in securitization trusts – impaired loans (net)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential mortgage loans held in securitization trusts – impaired loans (net)	$ 9,565	$ 8,976
Real estate owned held in residential securitization trusts
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Real estate owned held in residential securitization trusts	434	411
Level 1 | Residential mortgage loans held in securitization trusts – impaired loans (net)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential mortgage loans held in securitization trusts – impaired loans (net)	0	0
Level 1 | Real estate owned held in residential securitization trusts
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Real estate owned held in residential securitization trusts	0	0
Level 2 | Residential mortgage loans held in securitization trusts – impaired loans (net)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential mortgage loans held in securitization trusts – impaired loans (net)	0	0
Level 2 | Real estate owned held in residential securitization trusts
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Real estate owned held in residential securitization trusts	0	0
Level 3 | Residential mortgage loans held in securitization trusts – impaired loans (net)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential mortgage loans held in securitization trusts – impaired loans (net)	9,565	8,976
Level 3 | Real estate owned held in residential securitization trusts
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Real estate owned held in residential securitization trusts	$ 434	$ 411