Residential Mortgage Loans Held in Securitization Trusts (Net) and Real Estate Owned - Allowance for Loan Losses (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	$ 3,631	$ 2,989	$ 2,978
Provisions for loan losses	1,161	998	701
Transfer to real estate owned	0	(356)	(407)
Charge-offs	(1,393)	0	(283)
Balance at the end of period	$ 3,399	$ 3,631	$ 2,989