Loans - Ratings (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2016 USD ($) loan	Dec. 31, 2015 USD ($)
Investments in loans
Total gross loans	$ 6,031,123	$ 6,269,546
Total gross loans (as a percent)	100.00%	100.00%
Amount of loan impairment charges on individual loans held-for-investment	$ 0	$ 0
Rating 1
Investments in loans
Total gross loans	$ 3,070	664
Total gross loans (as a percent)	0.10%
Rating 1 | Maximum
Investments in loans
LTV (as a percent)	65.00%
Rating 2
Investments in loans
Total gross loans	$ 789,139	$ 675,678
Total gross loans (as a percent)	13.10%	10.80%
Rating 2 | Maximum
Investments in loans
LTV (as a percent)	70.00%
Rating 3
Investments in loans
Total gross loans	$ 4,660,465	$ 4,987,459
Total gross loans (as a percent)	77.30%	79.60%
Rating 3 | Maximum
Investments in loans
LTV (as a percent)	80.00%
Rating 4
Investments in loans
Total gross loans	$ 279,146	$ 401,880
Total gross loans (as a percent)	4.60%	6.40%
Allowance for loan losses as a percent of carrying amount	1.50%
Rating 4 | Minimum
Investments in loans
LTV (as a percent)	80.00%
Rating 4 | Maximum
Investments in loans
LTV (as a percent)	90.00%
Rating 5
Investments in loans
Total gross loans	$ 62,197
Total gross loans (as a percent)	1.00%
Allowance for loan losses as a percent of carrying amount	5.00%
Rating 5 | Minimum
Investments in loans
LTV (as a percent)	90.00%
N/A
Investments in loans
Total gross loans	$ 237,106	$ 203,865
Total gross loans (as a percent)	3.90%	3.20%
Total loans held-for-investment
Investments in loans
Total gross loans	$ 5,700,749	$ 5,979,108
Total loans held-for-investment | First Mortgages, excluding Cost Recovery Loans
Investments in loans
Total gross loans	4,538,986	4,723,852
Total loans held-for-investment | First Mortgages, excluding Cost Recovery Loans | Rating 1
Investments in loans
Total gross loans	3,070	664
Total loans held-for-investment | First Mortgages, excluding Cost Recovery Loans | Rating 2
Investments in loans
Total gross loans	606,666	496,372
Total loans held-for-investment | First Mortgages, excluding Cost Recovery Loans | Rating 3
Investments in loans
Total gross loans	3,728,584	3,979,247
Total loans held-for-investment | First Mortgages, excluding Cost Recovery Loans | Rating 4
Investments in loans
Total gross loans	200,666	247,569
Total loans held-for-investment | Subordinated mortgages
Investments in loans
Total gross loans	392,208	392,563
Total loans held-for-investment | Subordinated mortgages | Rating 2
Investments in loans
Total gross loans	86,069	88,857
Total loans held-for-investment | Subordinated mortgages | Rating 3
Investments in loans
Total gross loans	285,921	270,435
Total loans held-for-investment | Subordinated mortgages | Rating 4
Investments in loans
Total gross loans	20,218	33,271
Total loans held-for-investment | Mezzanine Loans
Investments in loans
Total gross loans	769,555	862,693
Total loans held-for-investment | Mezzanine Loans | Rating 2
Investments in loans
Total gross loans	96,404	90,449
Total loans held-for-investment | Mezzanine Loans | Rating 3
Investments in loans
Total gross loans	552,692	651,204
Total loans held-for-investment | Mezzanine Loans | Rating 4
Investments in loans
Total gross loans	58,262	121,040
Total loans held-for-investment | Mezzanine Loans | Rating 5
Investments in loans
Total gross loans	62,197
Loans held-for-sale
Investments in loans
Total gross loans	$ 237,106	203,865
Number of held-for-sale loans with the fair value option elected that were 90 days past due or greater or on nonaccrual status | loan	0
Loans held-for-sale | N/A
Investments in loans
Total gross loans	$ 237,106	203,865
Loans transferred as secured borrowings
Investments in loans
Total gross loans	93,268	86,573
Loans transferred as secured borrowings | Rating 3
Investments in loans
Total gross loans	$ 93,268	$ 86,573