Variable Interest Entities Carrying Value Components (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Residential Whole-Loans
Variable Interest Entity [Line Items]
Principal balance	$ 232,270	$ 187,765
Unamortized premium	2,021	1,311
Unamortized discount	(1,190)	(539)
Amortized cost	233,101	188,537
Gross unrealized gains	4,463	3,643
Gross unrealized losses	(141)	(44)
Fair value	237,423	192,136
Residential Bridge Loans, at Fair Value
Variable Interest Entity [Line Items]
Principal balance	63,802
Unamortized premium	293
Unamortized discount	(128)
Amortized cost	63,967
Gross unrealized gains	655
Gross unrealized losses	(96)
Fair value	64,526
Residential Bridge Loans, at Amortized Cost
Variable Interest Entity [Line Items]
Principal balance	42,066
Unamortized premium	122
Unamortized discount	(41)
Amortized cost	42,147
Securitized commercial loan
Variable Interest Entity [Line Items]
Principal balance	24,846	25,000
Amortized cost	24,846	25,000
Gross unrealized gains	30
Gross unrealized losses	0	(775)
Fair value	$ 24,876	$ 24,225