Fair Value of Financial Instruments Level 3 (Details) - Fair Value, Measurements, Recurring $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($)
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	$ 16,435
Unrealized (loss) gain included in OCI	(1,057)
Principal payments	(6,019)
Accretion	1,568
Balance at the end of the period	10,927
Non-Agency CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	14,903
Unrealized (loss) gain included in OCI	(1,055)
Principal payments	(5,747)
Accretion	1,568
Balance at the end of the period	9,669
Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period	1,532
Unrealized (loss) gain included in OCI	(2)
Principal payments	(272)
Accretion	0
Balance at the end of the period	$ 1,258