Investment Securities - AFS and Fair Value Option (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($) security	Dec. 31, 2015 USD ($) security	Dec. 31, 2014 USD ($)
RMBS | Available-for-sale
Investment Securities
Cost of third party management	$ 1,800	$ 900	$ 1,900
Estimated Fair Value
Securities with a loss less than 12 months	8,819	17,026
Securities with a loss greater than 12 months	957	653
Unrealized Losses
Securities with a loss less than 12 months	(90)	(180)
Securities with a loss greater than 12 months	$ (94)	$ (160)
Number of securities with unrealized loss position | security	3	5
Portion of securities with variable rate	$ 211,100	$ 122,700
CMBS | Fair value option
Unrealized Losses
Fair value of investment securities before consolidation of VIEs	1,000,000
Unpaid principal balance of investment securities before consolidation of VIEs	4,500,000
Fair value of investment securities before consolidation of VIEs eliminated against VIE liabilities	959,000
Purchases in which fair value option was elected	168,000	354,200	264,000
Purchase amount reflected as repayment of debt of consolidated VIEs	110,400	339,500	143,900
Sales in which fair value option was elected	54,400	15,500	121,400
Sale amount reflected as issuance of debt of consolidated VIEs	35,700	$ 9,100	$ 89,400
Portion of securities with variable rate	$ 0