Investment Securities - AFS and Fair Value Option (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2017 USD ($) security	Mar. 31, 2016 USD ($)	Dec. 31, 2016 USD ($)
RMBS
Unrealized Losses
Portion of securities with variable rate	$ 208,300
RMBS | Available-for-sale
Investment Securities
Cost of third party management	500	$ 400
Estimated Fair Value
Securities with a loss less than 12 months	1,601		$ 8,819
Securities with a loss greater than 12 months	954		957
Unrealized Losses
Securities with a loss less than 12 months	(17)		(90)
Securities with a loss greater than 12 months	$ (68)		(94)
Number of securities with unrealized loss position | security	3
Portion of securities with variable rate			$ 211,100
CMBS | Fair value option
Unrealized Losses
Fair value of investment securities before consolidation of VIEs	$ 1,000,000
Unpaid principal balance of investment securities before consolidation of VIEs	4,300,000
Purchases in which fair value option was elected	57,400	46,600
Purchase amount reflected as repayment of debt of consolidated VIEs	57,400	13,400
Sales in which fair value option was elected	15,200	600
Sale amount reflected as issuance of debt of consolidated VIEs	4,800	$ 600
CMBS, fair value option
Unrealized Losses
Portion of securities with variable rate	$ 0