Variable Interest Entities Summary of the Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Variable Interest Entity [Line Items]
Residential Whole-Loans, at fair value	$ 237,423	$ 192,136
Securitized commercial loan, at fair value	24,876	24,225
Investment related receivable	7,665	33,600
Accrued interest receivable	13,603	18,812
Total assets	3,886,906	3,156,016
Securitized debt, at fair value	10,945	10,659
Accrued interest payable	8,322	16,041
Accounts payable and accrued expenses	3,118	3,255
Total liabilities	3,420,868	2,725,534
VIE
Variable Interest Entity [Line Items]
Residential Whole-Loans, at fair value	237,423	192,136
Residential Bridge Loans	106,673	0
Securitized commercial loan, at fair value	24,876	24,225
Investment related receivable	7,665	1,241
Accrued interest receivable	3,358	1,622
Total assets	379,995	219,224
Securitized debt, at fair value	10,945	10,659
Accrued interest payable	70	85
Accounts payable and accrued expenses	189	2
Total liabilities	11,204	$ 10,746
Residential Bridge Loans, at Fair Value
Variable Interest Entity [Line Items]
Fair value	64,526
Amortized cost	63,967
Residential Bridge Loans, at Amortized Cost
Variable Interest Entity [Line Items]
Amortized cost	$ 42,147