Borrowings - Collateral Positions (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Collateral Positions
Accrued interest	$ 12,617	$ 9,790
Pledged Assets Including Accrued Interest	3,569,911	2,513,908
Assets Pledged- Fair Value	3,557,294	2,504,118
Due from counterparties	23,600	37,000
Due to counterparties	1,500	270
Securities received as collateral	306	357
Repurchase agreements
Collateral Positions
Amount of secured collateral	3,500,000	2,500,000
Repurchase agreements | Agency RMBS
Collateral Positions
Amount of secured collateral	690,255	1,465,384
Accrued interest	2,601	5,335
Pledged Assets Including Accrued Interest	692,856	1,470,719
Repurchase agreements | Non-Agency CMBS
Collateral Positions
Amount of secured collateral	208,062	358,919
Accrued interest	1,100	1,845
Pledged Assets Including Accrued Interest	209,162	360,764
Repurchase agreements | Non-Agency RMBS
Collateral Positions
Amount of secured collateral	58,127	308,165
Accrued interest	160	682
Pledged Assets Including Accrued Interest	58,287	308,847
Repurchase agreements | Whole-Loans and securitized commercial loan
Collateral Positions
Amount of secured collateral	237,423	205,702
Accrued interest	1,754	1,518
Pledged Assets Including Accrued Interest	239,177	207,220
Repurchase agreements | Residential Bridge Loans
Collateral Positions
Amount of secured collateral	106,673	0
Accrued interest	1,443	0
Pledged Assets Including Accrued Interest	108,116	0
Repurchase agreements | Other securities
Collateral Positions
Amount of secured collateral	89,823	67,762
Accrued interest	105	57
Pledged Assets Including Accrued Interest	89,928	67,819
Repurchase agreements | Cash
Collateral Positions
Pledged Assets Including Accrued Interest	23,591	36,986
Assets Pledged- Fair Value	23,591	36,986
VIE
Collateral Positions
Amount of secured collateral	106,673	0
VIE | Agency CMBS
Collateral Positions
Amount of secured collateral	2,143,340	61,200
Accrued interest	5,454	353
Pledged Assets Including Accrued Interest	$ 2,148,794	$ 61,553