Fair Value of Financial Instruments Level 3 (Details) - USD ($) $ in Thousands		3 Months Ended	9 Months Ended
		Sep. 30, 2020	Sep. 30, 2020
Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		$ 1,490	$ 1,690
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Other Comprehensive Income (Loss)	[1]	(72)	(246)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(102)	(308)
Balance at the end of the period		1,398	1,398
Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		7,366	9,405
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Earnings	[1]	(213)	288
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(651)	(2,170)
Balance at the end of the period		6,921	6,921
Interest Income | Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		82	262
Interest Income | Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		$ (7)	$ (26)

[1] Change in fair value for other non-Agency MBS is recorded as unrealized gain (loss) in “other comprehensive income”. Change in fair value for mortgage loans is recorded as unrealized gain (loss) in “fair value adjustments, net“ and the amount shown for the nine months ended September 30, 2020 is net of cumulative adjustment of $(548) made to the amortized cost as of December 31, 2019 as a result of the Company’s election of the fair value option for its mortgage loans effective January 1, 2020.