Fair Value of Financial Instruments Level 3 (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2020 USD ($) percent	Jan. 01, 2020 USD ($)
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Cumulative Effect of New Accounting Principle in Period of Adoption			$ (548)
Measurement Input, Prepayment Rate
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Debt Securities, Available-for-sale, Measurement Input | percent		0.10
Loans Held-for-sale, Measurement Input | percent		0.10
Measurement Input, Default Rate
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Debt Securities, Available-for-sale, Measurement Input | percent		0.02
Loans Held-for-sale, Measurement Input | percent		0.02
Measurement Input, Loss Severity
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Debt Securities, Available-for-sale, Measurement Input | percent		0.20
Loans Held-for-sale, Measurement Input | percent		0.20
Measurement Input, Discount Rate
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Debt Securities, Available-for-sale, Measurement Input | percent		0.070
Loans Held-for-sale, Measurement Input | percent		0.070
Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		$ 1,690
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Other Comprehensive Income (Loss)	[1]	(316)
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(439)
Balance at the end of the period		1,275
Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at the beginning of the period		9,405
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Gain (Loss) Included in Earnings	[1]	253
Fair Value, Measurement with Unobservable Inputs Reconciliation, Recurring Basis, Asset, Settlements		(2,854)
Balance at the end of the period		6,264
Interest Income | Non-Agency MBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		340
Interest Income | Loans Receivable
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Accretion (amortization)		$ (34)

[1] Change in fair value for other non-Agency MBS is recorded as unrealized gain (loss) in “other comprehensive income”. Change in fair value for mortgage loans is recorded as unrealized gain (loss) in “gain(loss) on investments, net“ and the amount shown for the year ended December 31, 2020 is net of a cumulative adjustment of $(548) made to the amortized cost as of December 31, 2019 as a result of the Company’s election of the fair value option for its mortgage loans effective January 1, 2020.