Investment Securities - AFS and Fair Value Option (Details) $ in Thousands	1 Months Ended	3 Months Ended		9 Months Ended
	Sep. 30, 2016 USD ($)	Sep. 30, 2017 USD ($) security	Sep. 30, 2016 USD ($)	Sep. 30, 2017 USD ($) security	Sep. 30, 2016 USD ($)	Dec. 31, 2016 USD ($)
Unrealized Losses
Settlement received on sales of investment securities	$ 14,900
RMBS
Unrealized Losses
Portion of securities with variable rate		$ 211,600		$ 211,600
RMBS | Available-for-sale
Investment Securities
Cost of third party management		500	$ 500	1,400	$ 1,200
Estimated Fair Value
Securities with a loss less than 12 months		10,463		10,463		$ 8,819
Securities with a loss greater than 12 months		645		645		957
Unrealized Losses
Securities with a loss less than 12 months		(80)		(80)		(90)
Securities with a loss greater than 12 months		$ (29)		$ (29)		(94)
Number of securities with unrealized loss position | security		3		3
Portion of securities with variable rate						$ 211,100
CMBS | Fair value option
Unrealized Losses
Fair value of investment securities before consolidation of VIEs		$ 1,000,000		$ 1,000,000
Unpaid principal balance of investment securities before consolidation of VIEs		4,100,000		4,100,000
Purchases in which fair value option was elected		30,900	25,600	92,600	126,900
Purchase amount reflected as repayment of debt of consolidated VIEs		19,100	25,600	80,800	69,300
Sales in which fair value option was elected		1,500	17,500	22,800	19,300
Sale amount reflected as issuance of debt of consolidated VIEs		1,500	$ 0	11,700	$ 600
CMBS, fair value option
Unrealized Losses
Portion of securities with variable rate		$ 0		$ 0