Investment Securities - Summary of Purchases and Sales (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	$ 0	$ 10,391	$ 0	$ 16,120
Sales	0	0	0	7,940
Principal collections	9,353	36,920	68,867	50,479
Redemptions	0		0
VIE eliminations
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	(106,433)	(185,433)	(133,766)	(222,386)
Sales	(30,684)	0	(42,288)	(24,376)
Principal collections	(16,089)	(12,266)	(30,570)	(36,989)
Redemptions	(11,976)		(11,976)
Held-to-maturity
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	0	10,391	0	16,120
Sales	0	0	0	0
Principal collections	879	30,713	52,569	37,351
Redemptions	0		0
RMBS | Available-for-sale
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	0	0	0	0
Sales	0	0	0	0
Principal collections	7,863	6,014	15,114	12,563
Redemptions	0		0
RMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	52,351	185,433	79,684	214,725
Sales	30,684	0	30,684	0
Principal collections	14,687	11,532	28,031	20,104
Redemptions	11,976		11,976
CMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Purchases/fundings	54,082	0	54,082	7,661
Sales	0	0	11,604	32,316
Principal collections	2,013	$ 927	3,723	$ 17,450
Redemptions	$ 0		$ 0