Fair Value of Financial Instruments Level 3 (Details) - Fair Value, Measurements, Recurring $ in Thousands	9 Months Ended
	Sep. 30, 2016 USD ($)
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	$ 16,435
Unrealized (loss) gain included in OCI	(408)
Principal payments	(3,405)
Accretion	790
Balance at the end of the period	13,412
Non-Agency CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	14,903
Unrealized (loss) gain included in OCI	(427)
Principal payments	(3,223)
Accretion	790
Balance at the end of the period	12,043
Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	1,532
Unrealized (loss) gain included in OCI	19
Principal payments	(182)
Accretion	0
Balance at the end of the period	$ 1,369