Secured Financing Agreements (Details) $ in Thousands, € in Millions	1 Months Ended		3 Months Ended			6 Months Ended
	Jun. 30, 2016 EUR (€) loan item	May 31, 2016 USD ($)	Jun. 30, 2016 USD ($) item	Mar. 31, 2016 USD ($) item	Jun. 30, 2015 USD ($)	Jun. 30, 2016 USD ($) item	Jun. 30, 2015 USD ($)	Apr. 30, 2016 USD ($)	Feb. 29, 2016 USD ($)	Dec. 31, 2015 USD ($)
Secured Financing Agreements
Unamortized premium (discount), net			$ 1,414			$ 1,414				$ (1,702)
Unamortized deferred financing costs			(32,588)			(32,588)				(38,336)
Carrying Value			$ 4,476,221			$ 4,476,221				3,980,699
Repayment of secured financings
Percentage of repurchase agreements for which margin calls are limited to collateral specific credit marks			62.00%			62.00%
Percentage of repurchase agreements containing margin call provisions that pertain to loans held-for-sale			33.00%			33.00%
Lender 2 Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size				$ 600,000					$ 500,000
Amount of increase in available borrowings				100,000
Lender 4 Repo 2 Facility
Secured Financing Agreements
Maximum borrowing capacity			$ 600,000			$ 600,000
Maximum facility size subject to certain conditions			1,000,000			1,000,000
Maximum allowed for conduit loan financing								$ 200,000
Lender 7 Secured Financing
Secured Financing Agreements
Maximum borrowing capacity			450,000			450,000
Maximum facility size subject to certain conditions			$ 650,000			$ 650,000
Floor interest rate (as a percent)						75.00%
CMBS Repo 1 Facility
Secured Financing Agreements
Maturity period						11 months
CMBS Repo 2 Facility
Secured Financing Agreements
Number of first mortgage loans financed | loan	1
Number of first mortgage loan portfolios financed | item	1
Total amount of financing | €	€ 124.1
CMBS Repo 2 Facility | EURIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.00%
CMBS Repo 3 Facility
Secured Financing Agreements
Maturity period						12 months
RMBS Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size		$ 185,000						$ 125,000
Maturity period		270 days
Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Interest rate (as a percent)			3.50%			3.50%
Maximum Facility Size			$ 32,200			$ 32,200
Number of mortgage facilities executed | item						4
Investing and Servicing Segment Property Mortgages | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.25%
Investing and Servicing Segment Property Mortgages | Weighted-average
Secured Financing Agreements
Maturity period						6 years 1 month 6 days
Woodstar Portfolio Mortgages
Secured Financing Agreements
Principal Amount			$ 18,600			$ 18,600
Number of mortgage facilities assumed | item			2
Repayment of secured financings
Total			$ 18,600			18,600
Woodstar Portfolio Government Financing
Secured Financing Agreements
Principal Amount			129,200	$ 2,500		$ 129,200
Number of federal, state and county sponsored mortgage facilities assumed | item				1		17
Repayment of secured financings
Total			129,200	$ 2,500		$ 129,200
Secured financing agreements
Secured Financing Agreements
Pledged Asset Carrying Value			8,719,135			8,719,135
Maximum Facility Size			6,989,601			6,989,601
Principal Amount			4,507,395			4,507,395				4,020,737
Repayment of secured financings
2016 (remainder of)			180,865			180,865
2017			858,000			858,000
2018			1,025,315			1,025,315
2019			636,657			636,657
2020			1,279,723			1,279,723
Thereafter			526,835			526,835
Total			4,507,395			4,507,395				4,020,737
Amortization of deferred financing costs from secured financing agreements included in interest expense			4,300		$ 3,500	8,200	$ 7,000
Secured financing agreements | Lender 1 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			2,179,007			2,179,007
Maximum Facility Size			1,600,000			1,600,000
Principal Amount			1,490,949			1,490,949				975,735
Repayment of secured financings
Total			1,490,949			$ 1,490,949				975,735
Secured financing agreements | Lender 1 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.85%
Secured financing agreements | Lender 1 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						5.25%
Secured financing agreements | Lender 2 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			300,368			$ 300,368
Maximum Facility Size			500,000			500,000
Principal Amount			238,479			238,479				233,705
Repayment of secured financings
Total			238,479			$ 238,479				233,705
Secured financing agreements | Lender 2 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.75%
Secured financing agreements | Lender 2 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.75%
Secured financing agreements | Lender 3 Repo I Facility
Secured Financing Agreements
Pledged Asset Carrying Value			112,022			$ 112,022
Maximum Facility Size			79,325			79,325
Principal Amount			79,325			79,325				131,997
Repayment of secured financings
Total			79,325			$ 79,325				131,997
Secured financing agreements | Lender 3 Repo I Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.50%
Secured financing agreements | Lender 3 Repo I Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.85%
Secured financing agreements | Lender 4 Repo 1 Facility
Secured Financing Agreements
Principal Amount										309,498
Repayment of secured financings
Total										309,498
Secured financing agreements | Lender 4 Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.00%
Secured financing agreements | Lender 4 Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			253,813			$ 253,813
Maximum Facility Size			1,000,000			1,000,000
Principal Amount			164,940			164,940
Repayment of secured financings
2016 (remainder of)			62,900			62,900
Total			164,940			$ 164,940
Secured financing agreements | Lender 4 Repo 2 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.00%
Secured financing agreements | Lender 4 Repo 2 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.50%
Secured financing agreements | Lender 6 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			431,821			$ 431,821
Maximum Facility Size			500,000			500,000
Principal Amount			288,149			288,149				491,263
Repayment of secured financings
Total			288,149			$ 288,149				491,263
Secured financing agreements | Lender 6 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.50%
Secured financing agreements | Lender 6 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						3.00%
Secured financing agreements | Lender 7 Secured Financing
Secured Financing Agreements
Pledged Asset Carrying Value			108,120			$ 108,120
Maximum Facility Size			650,000			$ 650,000
Principal Amount										38,055
Repayment of secured financings
Total										38,055
Secured financing agreements | Lender 7 Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.75%
Secured financing agreements | Conduit Repo 1 Facility
Secured Financing Agreements
Principal Amount										80,741
Repayment of secured financings
Total										80,741
Secured financing agreements | Conduit Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			42,612			$ 42,612
Maximum Facility Size			150,000			150,000
Principal Amount			31,594			31,594
Repayment of secured financings
Total			31,594			$ 31,594
Secured financing agreements | Conduit Repo 2 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.10%
Secured financing agreements | Conduit Repo 3 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			9,096			$ 9,096
Maximum Facility Size			150,000			150,000
Principal Amount			6,825			6,825				66,041
Repayment of secured financings
Total			6,825			$ 6,825				66,041
Secured financing agreements | Conduit Repo 3 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.10%
Secured financing agreements | Conduit Repo 4 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			62,377			$ 62,377
Maximum Facility Size			100,000			100,000
Principal Amount			46,612			46,612
Repayment of secured financings
2016 (remainder of)			46,600			46,600
Total			46,612			$ 46,612
Secured financing agreements | Conduit Repo 4 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.25%
Secured financing agreements | CMBS Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			32,800			$ 32,800
Maximum Facility Size			21,354			21,354
Principal Amount			21,354			21,354
Repayment of secured financings
Total			21,354			$ 21,354
Secured financing agreements | CMBS Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.90%
Secured financing agreements | CMBS Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			339,132			$ 339,132
Maximum Facility Size			247,192			247,192
Principal Amount			247,192			247,192				120,850
Repayment of secured financings
Total			247,192			$ 247,192				120,850
Secured financing agreements | CMBS Repo 2 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.00%
Secured financing agreements | CMBS Repo 2 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.70%
Secured financing agreements | CMBS Repo 3 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			409,685			$ 409,685
Maximum Facility Size			287,467			287,467
Principal Amount			287,467			287,467				243,434
Repayment of secured financings
Total			287,467			$ 287,467				243,434
Secured financing agreements | CMBS Repo 3 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.40%
Secured financing agreements | CMBS Repo 3 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.85%
Secured financing agreements | RMBS Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value			157,641			$ 157,641
Maximum Facility Size			185,000			185,000
Principal Amount			91,144			$ 91,144				2,000
Current maturity period, relative to when the buyer delivers notice to the seller						270 days
Repayment of secured financings
Total			91,144			$ 91,144				2,000
Secured financing agreements | RMBS Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.90%
Secured financing agreements | Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value			150,199			$ 150,199
Maximum Facility Size			124,061			124,061
Principal Amount			118,163			118,163				82,964
Repayment of secured financings
Total			118,163			118,163				82,964
Secured financing agreements | Ireland Portfolio Mortgage
Secured Financing Agreements
Pledged Asset Carrying Value			483,814			483,814
Maximum Facility Size			326,558			326,558
Principal Amount			326,558			326,558				319,322
Repayment of secured financings
Total			326,558			$ 326,558				319,322
Secured financing agreements | Ireland Portfolio Mortgage | EURIBOR
Secured Financing Agreements
Pricing margin (as a percent)						1.69%
Secured financing agreements | Woodstar Portfolio Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value			380,690			$ 380,690
Maximum Facility Size			267,114			267,114
Principal Amount			267,114			267,114				248,630
Repayment of secured financings
Total			$ 267,114			$ 267,114				248,630
Secured financing agreements | Woodstar Portfolio Mortgages | Minimum
Secured Financing Agreements
Interest rate (as a percent)			3.72%			3.72%
Secured financing agreements | Woodstar Portfolio Mortgages | Maximum
Secured Financing Agreements
Interest rate (as a percent)			7.46%			7.46%
Secured financing agreements | Woodstar Portfolio Mortgages | Weighted-average
Secured Financing Agreements
Interest rate (as a percent)			3.99%			3.99%
Secured financing agreements | Woodstar Portfolio Government Financing
Secured Financing Agreements
Pledged Asset Carrying Value			$ 318,501			$ 318,501
Maximum Facility Size			137,394			137,394
Principal Amount			137,394			137,394				8,982
Repayment of secured financings
Total			$ 137,394			$ 137,394				8,982
Secured financing agreements | Woodstar Portfolio Government Financing | Minimum
Secured Financing Agreements
Interest rate (as a percent)			1.00%			1.00%
Secured financing agreements | Woodstar Portfolio Government Financing | Maximum
Secured Financing Agreements
Interest rate (as a percent)			5.00%			5.00%
Secured financing agreements | Term Loan
Secured Financing Agreements
Pledged Asset Carrying Value			$ 2,937,230			$ 2,937,230
Maximum Facility Size			654,886			654,886
Principal Amount			654,886			654,886				658,270
Repayment of secured financings
Total			654,886			$ 654,886				658,270
Secured financing agreements | Term Loan | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.75%
Secured financing agreements | FHLB Advances
Secured Financing Agreements
Pledged Asset Carrying Value			10,207			$ 10,207
Maximum Facility Size			9,250			9,250
Principal Amount			9,250			9,250				9,250
Repayment of secured financings
Total			9,250			$ 9,250				9,250
Secured financing agreements | FHLB Advances | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)						0.37%
Repurchase Agreements
Secured Financing Agreements
Pledged Asset Carrying Value			2,994,030			$ 2,994,030				2,655,264
Principal Amount			2,994,030			2,994,030
Repayment of secured financings
2016 (remainder of)			166,552			166,552
2017			827,610			827,610
2018			994,507			994,507
2019			617,347			617,347
2020			308,421			308,421
Thereafter			79,593			79,593
Total			2,994,030			2,994,030
Repurchase Agreements | Loan held for investment
Secured Financing Agreements
Pledged Asset Carrying Value			2,198,902			2,198,902				2,142,198
Repurchase Agreements | Loans held-for-sale
Secured Financing Agreements
Pledged Asset Carrying Value			147,971			147,971				146,782
Repurchase Agreements | Investment securities.
Secured Financing Agreements
Pledged Asset Carrying Value			647,157			647,157				$ 366,284
Other Secured Financing
Secured Financing Agreements
Principal Amount			1,513,365			1,513,365
Repayment of secured financings
2016 (remainder of)			14,313			14,313
2017			30,390			30,390
2018			30,808			30,808
2019			19,310			19,310
2020			971,302			971,302
Thereafter			447,242			447,242
Total			$ 1,513,365			$ 1,513,365