Financings - Collateral Positions (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	$ 27,571	$ 34,897
Accrued Interest	9,888	13,449
Assets Pledged and Accrued Interest	1,849,521	3,272,754
Assets pledged- fair value	1,839,633	3,259,305
VIE
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	27,571	34,897
Agency CMBS | VIE
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	14,442	356,687
Accrued Interest	354	1,336
Assets Pledged and Accrued Interest	14,796	358,023
Repurchase agreements
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	1,700,000	3,200,000
Repurchase agreements | Subtotal Agency
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	457,400	1,400,230
Accrued Interest	1,036	3,916
Assets Pledged and Accrued Interest	458,436	1,404,146
Repurchase agreements | Non-Agency RMBS
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	239,351	246,797
Accrued Interest	1,085	951
Assets Pledged and Accrued Interest	240,436	247,748
Repurchase agreements | Non-Agency CMBS
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	26,296	45,816
Accrued Interest	336	414
Assets Pledged and Accrued Interest	26,632	46,230
Repurchase agreements | Whole-Loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	576,769	529,495
Accrued Interest	4,732	3,704
Assets Pledged and Accrued Interest	581,501	533,199
Repurchase agreements | Residential Bridge Loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	27,571	34,897
Accrued Interest	359	471
Assets Pledged and Accrued Interest	27,930	35,368
Repurchase agreements | Commercial Loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	320,308	350,213
Accrued Interest	1,723	1,855
Assets Pledged and Accrued Interest	322,031	352,068
Repurchase agreements | Securitized commercial loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	39,649	171,640
Accrued Interest	127	674
Assets Pledged and Accrued Interest	39,776	172,314
Repurchase agreements | Other securities
Financial Instruments Owned and Pledged as Collateral [Line Items]
MBS pledged for borrowings under repurchase agreements	47,307	80,031
Accrued Interest	136	128
Assets Pledged and Accrued Interest	47,443	80,159
Repurchase agreements | Cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Assets Pledged and Accrued Interest	90,540	43,499
Assets pledged- fair value	$ 90,540	$ 43,499