Repurchase Agreements (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2017	Mar. 31, 2016	Dec. 31, 2016
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 2,825,945		$ 2,898,952
Secured borrowing, weighted average interest rate	1.15%		1.03%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 3,070,075		$ 3,149,839
Weighted average maturity remaining	18 days	20 days
Securitization financing bond
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 4,370		$ 4,803
Secured borrowing, weighted average interest rate	2.21%		2.00%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 4,850		$ 5,278
RMBS | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 1,020,367		$ 1,157,302
Secured borrowing, weighted average interest rate	0.90%		0.82%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 1,055,641		$ 1,191,147
RMBS | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 20,800		$ 26,149
Secured borrowing, weighted average interest rate	2.18%		1.98%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 24,944		$ 31,952
CMBS | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 1,068,251		$ 1,005,726
Secured borrowing, weighted average interest rate	0.90%		0.82%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 1,146,915		$ 1,095,002
CMBS | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 65,327		$ 66,881
Secured borrowing, weighted average interest rate	1.85%		1.63%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 77,198		$ 77,840
CMBS IO | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 344,828		$ 346,892
Secured borrowing, weighted average interest rate	1.82%		1.57%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 406,244		$ 407,481
CMBS IO | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Transfers Accounted for as Secured Borrowings, Associated Liabilities, Carrying Amount	$ 302,002		$ 291,199
Secured borrowing, weighted average interest rate	1.88%		1.67%
Pledged Financial Instruments, Not Separately Reported, Securities for Repurchase Agreements	$ 354,283		$ 341,139