Investment Securities - Summary of Purchases and Sales (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2021	Sep. 30, 2020	Sep. 30, 2021	Sep. 30, 2020
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	$ 0	$ 6,288	$ 0	$ 22,408
Sales	0		0	7,940
Principal collections	8,988	17,478	77,855	67,957
Redemptions	0		0
VIE eliminations
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	(41,392)	(43,083)	(175,158)	(265,469)
Sales	(27,111)		(69,399)	(24,376)
Principal collections	(14,555)	(20,185)	(45,125)	(57,174)
Redemptions	(26,753)		(38,729)
Held-to-maturity
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	0	6,288	0	22,408
Sales	0		0	0
Principal collections	752	11,203	53,321	48,554
Redemptions	0		0
RMBS | Available-for-sale
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	0	0	0	0
Sales	0		0	0
Principal collections	7,881	6,063	22,995	18,626
Redemptions	0		0
RMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	33,009	43,083	112,693	257,808
Sales	0		30,684	0
Principal collections	14,204	12,132	42,235	32,236
Redemptions	26,753		38,729
CMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	8,383	0	62,465	7,661
Sales	27,111		38,715	32,316
Principal collections	706	$ 8,265	4,429	$ 25,715
Redemptions	$ 0		$ 0