Variable Interest Entities (Details) $ in Thousands	1 Months Ended	12 Months Ended
	Nov. 30, 2015 USD ($)	Dec. 31, 2017 USD ($) loan entity	Dec. 31, 2016 USD ($) loan entity
Variable Interest Entity [Line Items]
Repurchase agreements		$ 3,251,686	$ 2,155,644
Securitized commercial loan, at fair value		24,876	24,225
Securitized debt, at fair value		$ 10,945	$ 10,659
Residential Whole-Loans
Variable Interest Entity [Line Items]
Number of VIEs, as primary beneficiary | entity		2
Number of Loans | loan		616	475
CMSC Trust
Variable Interest Entity [Line Items]
Securitized commercial loan, at fair value		$ 24,800
Trust certificates issued		$ 24,800
Variable Interest Entity, Not Primary Beneficiary, Aggregated Disclosure
Variable Interest Entity [Line Items]
Number of trusts | entity		3	3
VIE
Variable Interest Entity [Line Items]
Securitized commercial loan, at fair value		$ 24,876	$ 24,225
Securitized debt, at fair value		$ 10,945	$ 10,659
Debt Instrument, Interest Rate, Effective Percentage		8.90%	8.90%
VIE | Residential Whole-Loans
Variable Interest Entity [Line Items]
Repurchase agreements		$ 189,300	$ 154,400
Mortgage loans on real estate, number of loans | loan		1
Principal Amount of Loans Subject to Delinquent Principal or Interest		$ 579
Mortgage loans on real estate, fair value amount of delinquent loans		$ 570
Mortgage loans on real estate, number of loans, nonperforming, percentage		0.20%
VIE | Residential Bridge Loans
Variable Interest Entity [Line Items]
Repurchase agreements		$ 100,200
Mortgage loans on real estate, number of loans | loan		9
Principal Amount of Loans Subject to Delinquent Principal or Interest		$ 1,800
Mortgage loans on real estate, number of loans, nonperforming, percentage		1.70%
Original LTV, collateral dependent		74.00%
Number of Loans | loan		340
VIE | Securitized commercial loan
Variable Interest Entity [Line Items]
Number of Loans | loan		1
CMSC Trust | Securitized commercial loan
Variable Interest Entity [Line Items]
Amount acquired, eliminated in consolidation	$ 14,000	$ 13,900
Variable interest entity, fair value		13,900
Securitized debt, at fair value		10,900
VIE | Securitized debt | Securitized commercial loan
Variable Interest Entity [Line Items]
Securitized debt, at fair value		$ 10,900	$ 10,700