Real Estate Securities - Available for Sale Securities by Collateral Type (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Investment Holdings [Line Items]
Available-for-sale securities	$ 572,752	$ 829,245
Senior Securities
Investment Holdings [Line Items]
Available-for-sale securities	136,546	279,251
Senior Securities | Prime
Investment Holdings [Line Items]
Available-for-sale securities	128,843	210,993
Senior Securities | Non-prime
Investment Holdings [Line Items]
Available-for-sale securities	7,703	68,258
Re-REMIC
Investment Holdings [Line Items]
Available-for-sale securities	85,479	165,064
Subordinate Securities
Investment Holdings [Line Items]
Available-for-sale securities	350,727	384,930
Subordinate Securities | Prime mezzanine
Investment Holdings [Line Items]
Available-for-sale securities	163,715	224,624
Subordinate Securities | Prime subordinate
Investment Holdings [Line Items]
Available-for-sale securities	$ 187,012	$ 160,306