Fair Value of Financial Instruments Level 3 (Details) - USD ($) $ in Thousands		3 Months Ended	6 Months Ended
		Jun. 30, 2020	Jun. 30, 2020
Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		$ 1,562	$ 1,690
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Other Comprehensive Income (Loss)	[1]	(57)	(174)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(101)	(206)
Balance at the end of the period		1,490	1,490
Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		7,922	9,405
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Earnings	[1]	(359)	501
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(904)	(1,519)
Balance at the end of the period		7,366	7,366
Interest Income | Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		86	180
Interest Income | Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		$ (11)	$ (19)

[1] Change in fair value for other non-Agency MBS is recorded as unrealized gain (loss) in “other comprehensive income”. Change in fair value for mortgage loans is recorded as unrealized gain (loss) in “fair value adjustments, net“ and the amount shown for the six months ended June 30, 2020 is net of cumulative adjustment of $(548) made to the amortized cost as of December 31, 2019 as a result of the Company’s election of the fair value option for its mortgage loans effective January 1, 2020.