Summary of Significant Accounting Policies - Narrative (Details)		1 Months Ended	3 Months Ended		6 Months Ended
	May 16, 2016	May 31, 2015	Jun. 30, 2016 USD ($) securitization	Jun. 30, 2015 USD ($)	Jun. 30, 2016 USD ($) segment securitization	Jun. 30, 2015 USD ($)	Dec. 31, 2015 USD ($) securitization
Accounts, Notes, Loans and Financing Receivable [Line Items]
Goodwill			$ 24,782,000		$ 24,782,000		$ 0
Residential mortgage loans, delinquency period					90 days
Restricted cash			55,700,000		$ 55,700,000		20,800,000
Interest receivable			23,600,000		23,600,000		24,600,000
Deferred finance costs, gross			1,800,000		1,800,000		1,000,000
Employer contributions			100,000	$ 0	$ 100,000	$ 0
Number of operating segments | segment					1
Number of reportable segments | segment					1
Held in Agency IO Portfolio for Trading Purposes
Accounts, Notes, Loans and Financing Receivable [Line Items]
Restricted cash			39,700,000		$ 39,700,000		11,600,000
Held by Counterparties as Collateral for Hedging Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Restricted cash			10,600,000		10,600,000		8,200,000
Distressed Residential Mortgage Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Deferred finance costs, gross			$ 1,400,000		$ 1,400,000		$ 2,300,000
Consolidated K-Series | K-Series
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of securitizations | securitization			5		5		5
Receivables and Other Assets
Accounts, Notes, Loans and Financing Receivable [Line Items]
Real estate under development			$ 15,600,000		$ 15,600,000		$ 0
Equity method investment			10,700,000		10,700,000		20,100,000
Receivables and Other Assets | Real Estate Loan | Mezzanine Loan and Preferred Equity Investments | Residential
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, net			75,300,000		75,300,000		44,200,000
Equity Method Investments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Equity method investment			63,100,000		63,100,000		67,600,000
Equity Method Investments | Receivables and Other Assets
Accounts, Notes, Loans and Financing Receivable [Line Items]
Equity method investment			$ 63,100,000		$ 63,100,000		$ 67,600,000
Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Finite-Lived Intangible Asset, Useful Life	1 year
Repurchase agreements, expiration period					12 months
Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Finite-Lived Intangible Asset, Useful Life	15 years
Repurchase agreements, expiration period					15 months
Performance Share Awards
Accounts, Notes, Loans and Financing Receivable [Line Items]
Award vesting period		3 years