Secured Financing Agreements (Details) $ in Thousands, £ in Millions	1 Months Ended						12 Months Ended
	Dec. 31, 2016 USD ($) item	Nov. 30, 2016 GBP (£)	Sep. 30, 2016 USD ($)	Aug. 31, 2016 USD ($)	May 31, 2016 USD ($)	Jan. 31, 2016	Dec. 31, 2016 USD ($) item	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)	Dec. 16, 2016 USD ($)	Nov. 30, 2016 USD ($)	Apr. 30, 2016 USD ($)	Mar. 31, 2016 USD ($)	Feb. 29, 2016 USD ($)
Secured Financing Agreements
Principal Amount	$ 2,053,229						$ 2,053,229	$ 1,372,594
Unamortized premium (discount), net	2,640						2,640	(1,702)
Unamortized deferred financing costs	(45,732)						(45,732)	(38,336)
Carrying Value	4,154,126						4,154,126	3,980,699
Repayment of secured financings
Total	$ 2,053,229						$ 2,053,229	1,372,594
Percentage of repurchase agreements for which margin calls are limited to collateral specific credit marks	41.00%						41.00%
Percentage of repurchase agreements containing margin call provisions that pertain to loans held-for-sale	18.00%						18.00%
Lender 1 Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size	$ 1,800,000						$ 1,800,000
Maximum borrowing capacity			$ 1,800,000	$ 1,600,000
Maximum facility size subject to certain conditions	2,000,000		2,000,000				2,000,000
Lender 2 Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size													$ 600,000	$ 500,000
Lender 4 Repo 2 Facility
Secured Financing Agreements
Maximum Facility Size	600,000						600,000					$ 200,000
Maximum facility size subject to certain conditions	$ 1,000,000						1,000,000
Lender 6 Repo 2 Facility
Secured Financing Agreements
Maximum Facility Size | £		£ 98.5
Maturity period		3 years
Repayment of secured financings
Extended term / option		1 year
Lender 6 Repo 2 Facility | Lending Segment
Secured Financing Agreements
Mortgage loan collateral | £		£ 142.5
Lender 6 Repo 2 Facility | GBP LIBOR
Secured Financing Agreements
Pricing margin (as a percent)		2.75%
Lender 9 Repo 1 Facility
Secured Financing Agreements
Pricing margin (as a percent)	1.65%
Maturity period	1 year
Repayment of secured financings
Extended term / option	1 year
Lender 9 Repo 1 Facility | Lending Segment
Secured Financing Agreements
Mortgage loan collateral	$ 378,100
Lender 7 Secured Financing
Secured Financing Agreements
Maximum Facility Size	450,000						450,000
Maximum facility size subject to certain conditions	650,000						$ 650,000
Lender 8 Secured Financing
Secured Financing Agreements
Principal Amount				$ 75,000
Maturity period				3 years
Repayment of secured financings
Total				$ 75,000
Lender 8 Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)				4.00%
Conduit Repo 4 Facility
Secured Financing Agreements
Maximum Facility Size													$ 100,000
MBS Repo 1 Facility
Secured Financing Agreements
Maturity period							11 months
MBS Repo 2 Facility | EURIBOR
Secured Financing Agreements
Pricing margin (as a percent)						2.00%
MBS Repo 3 Facility
Secured Financing Agreements
Maturity period							12 months
MBS Repo 4 Facility
Secured Financing Agreements
Maximum Facility Size			225,000		$ 185,000							$ 125,000
Amount of facility allowed to be utilized for financing of CMBS			$ 50,000
Maturity period					270 days
Current maturity period, relative to when the buyer delivers notice to the seller							270 days
Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Maximum Facility Size	$ 75,600						$ 75,600
Number of mortgage facilities executed | item							8
Number of facilities carrying floating interest rates | item	4						4
Investing and Servicing Segment Property Mortgages | Weighted-average
Secured Financing Agreements
Interest rate (as a percent)	3.53%						3.53%
Maturity period							4 years
Investing and Servicing Segment Property Mortgages | Weighted-average | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.27%
Woodstar Portfolio Government Financing
Secured Financing Agreements
Principal Amount	$ 135,600						$ 135,600
Number of federal, state and county sponsored mortgage facilities assumed | item							22
Repayment of secured financings
Total	135,600						$ 135,600
Woodstar Portfolio Government Financing | Additional Mortgage Facilities Acquired
Secured Financing Agreements
Interest rate (as a percent)			3.97%
Principal Amount	18,600		$ 28,100				$ 18,600
Maturity period			10 years
Number of mortgage facilities assumed | item							2
Repayment of secured financings
Total	18,600		$ 28,100				$ 18,600
Medical Office Portfolio Mortgages
Secured Financing Agreements
Principal Amount	$ 491,200						$ 491,200
Floor interest rate (as a percent)							0.25%
Maturity period	5 years
Number of mortgage facilities executed | item	2
Repayment of secured financings
Total	$ 491,200						$ 491,200
Number of extension options | item	2
Extended term / option	12 months
Medical Office Portfolio Mortgages | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.50%
Term Loan A
Secured Financing Agreements
Maximum Facility Size	$ 300,000						300,000
Maturity period	4 years
Repayment of secured financings
Number of extension options | item	2
Extended term / option	6 months
Term Loan A | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.25%
Term Loan B
Secured Financing Agreements
Carrying Value											$ 653,200
Repayment of secured financings
Write-off of deferred financing costs and unamortized discount	$ 8,200
Revolving Secured Financing
Secured Financing Agreements
Maximum Facility Size	$ 100,000						$ 100,000
Maturity period	4 years
Repayment of secured financings
Number of extension options | item	2
Extended term / option	6 months
Revolving Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.25%
2021 Senior Notes
Secured Financing Agreements
Interest rate (as a percent)	5.00%						5.00%			5.00%
Principal Amount	$ 700,000						$ 700,000
Amount issued										$ 700,000
Repayment of secured financings
Total	700,000						700,000
Secured financing agreements
Secured Financing Agreements
Pledged Asset Carrying Value	7,912,206						7,912,206
Maximum Facility Size	8,193,955						8,193,955
Principal Amount	4,197,218						4,197,218	4,020,737
Carrying Value	4,154,126						4,154,126	3,980,699
Repayment of secured financings
2017	681,480						681,480
2018	676,747						676,747
2019	632,910						632,910
2020	762,330						762,330
2021	397,031						397,031
Thereafter	1,046,720						1,046,720
Total	4,197,218						4,197,218	4,020,737
Amortization of deferred financing costs from secured financing agreements included in interest expense							16,200	14,200	$ 11,300
Secured financing agreements | Lender 1 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	1,645,064						1,645,064
Maximum Facility Size	2,000,000						2,000,000
Principal Amount	944,712						944,712	975,735
Repayment of secured financings
2017	142,900						142,900
Total	944,712						$ 944,712	975,735
Secured financing agreements | Lender 1 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.75%
Secured financing agreements | Lender 1 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							5.75%
Secured financing agreements | Lender 2 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	387,528						$ 387,528
Maximum Facility Size	500,000						500,000
Principal Amount	132,941						132,941	233,705
Repayment of secured financings
Total	132,941						$ 132,941	233,705
Secured financing agreements | Lender 2 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.75%
Secured financing agreements | Lender 2 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.75%
Secured financing agreements | Lender 3 Repo I Facility
Secured Financing Agreements
Pledged Asset Carrying Value	110,401						$ 110,401
Maximum Facility Size	78,288						78,288
Principal Amount	78,288						78,288	131,997
Repayment of secured financings
Total	78,288						$ 78,288	131,997
Secured financing agreements | Lender 3 Repo I Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.50%
Secured financing agreements | Lender 3 Repo I Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.85%
Secured financing agreements | Lender 4 Repo 1 Facility
Secured Financing Agreements
Principal Amount								309,498
Repayment of secured financings
Total								309,498
Secured financing agreements | Lender 4 Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	484,072						$ 484,072
Maximum Facility Size	1,000,000						1,000,000
Principal Amount	166,394						166,394
Repayment of secured financings
Total	166,394						$ 166,394
Secured financing agreements | Lender 4 Repo 2 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.00%
Secured financing agreements | Lender 4 Repo 2 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.50%
Secured financing agreements | Lender 6 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	376,953						$ 376,953
Maximum Facility Size	500,000						500,000
Principal Amount	182,586						182,586	491,263
Repayment of secured financings
Total	182,586						$ 182,586	491,263
Secured financing agreements | Lender 6 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.50%
Secured financing agreements | Lender 6 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.75%
Secured financing agreements | Lender 6 Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	173,621						$ 173,621
Maximum Facility Size	121,509						121,509
Principal Amount	121,509						121,509
Repayment of secured financings
Total	121,509						$ 121,509
Secured financing agreements | Lender 6 Repo 2 Facility | GBP LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.75%
Secured financing agreements | Lender 9 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	378,152						$ 378,152
Maximum Facility Size	283,575						283,575
Principal Amount	283,575						283,575
Repayment of secured financings
Total	283,575						$ 283,575
Secured financing agreements | Lender 9 Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.65%
Secured financing agreements | Lender 7 Secured Financing
Secured Financing Agreements
Pledged Asset Carrying Value	86,650						$ 86,650
Maximum Facility Size	650,000						$ 650,000
Principal Amount								38,055
Repayment of secured financings
Total								38,055
Secured financing agreements | Lender 7 Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.75%
Secured financing agreements | Lender 8 Secured Financing
Secured Financing Agreements
Pledged Asset Carrying Value	66,243						$ 66,243
Maximum Facility Size	75,000						75,000
Principal Amount	43,555						43,555
Repayment of secured financings
Total	43,555						$ 43,555
Secured financing agreements | Lender 8 Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							4.00%
Secured financing agreements | Conduit Repo 1 Facility
Secured Financing Agreements
Principal Amount								80,741
Repayment of secured financings
Total								80,741
Secured financing agreements | Conduit Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	20,035						$ 20,035
Maximum Facility Size	150,000						150,000
Principal Amount	14,944						14,944
Repayment of secured financings
Total	14,944						$ 14,944
Secured financing agreements | Conduit Repo 2 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.25%
Secured financing agreements | Conduit Repo 3 Facility
Secured Financing Agreements
Maximum Facility Size	150,000						$ 150,000
Principal Amount								66,041
Repayment of secured financings
Total								66,041
Secured financing agreements | Conduit Repo 3 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.10%
Secured financing agreements | Conduit Repo 4 Facility
Secured Financing Agreements
Maximum Facility Size	100,000						$ 100,000
Secured financing agreements | Conduit Repo 4 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.25%
Secured financing agreements | MBS Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	31,840						$ 31,840
Maximum Facility Size	21,052						21,052
Principal Amount	21,052						21,052
Repayment of secured financings
Total	21,052						$ 21,052
Secured financing agreements | MBS Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.90%
Secured financing agreements | MBS Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	329,667						$ 329,667
Maximum Facility Size	239,434						239,434
Principal Amount	239,434						239,434	120,850
Repayment of secured financings
2017	109,500						109,500
Total	239,434						$ 239,434	120,850
Secured financing agreements | MBS Repo 2 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.00%
Secured financing agreements | MBS Repo 2 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.95%
Secured financing agreements | MBS Repo 3 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	411,173						$ 411,173
Maximum Facility Size	285,209						285,209
Principal Amount	285,209						285,209	243,434
Repayment of secured financings
2017	285,200						285,200
Total	285,209						$ 285,209	243,434
Secured financing agreements | MBS Repo 3 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.37%
Secured financing agreements | MBS Repo 3 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.00%
Secured financing agreements | MBS Repo 4 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	188,670						$ 188,670
Maximum Facility Size	225,000						225,000
Principal Amount	5,633						5,633	2,000
Repayment of secured financings
Total	5,633						$ 5,633	2,000
Secured financing agreements | MBS Repo 4 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.20%
Secured financing agreements | MBS Repo 4 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.90%
Secured financing agreements | Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value	218,156						$ 218,156
Maximum Facility Size	168,811						168,811
Principal Amount	164,611						164,611	82,964
Repayment of secured financings
Total	164,611						164,611	82,964
Secured financing agreements | Ireland Portfolio Mortgage
Secured Financing Agreements
Pledged Asset Carrying Value	450,158						450,158
Maximum Facility Size	309,246						309,246
Principal Amount	309,246						309,246	319,322
Repayment of secured financings
Total	309,246						$ 309,246	319,322
Secured financing agreements | Ireland Portfolio Mortgage | EURIBOR
Secured Financing Agreements
Pricing margin (as a percent)							1.69%
Secured financing agreements | Woodstar Portfolio Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value	376,653						$ 376,653
Maximum Facility Size	276,748						276,748
Principal Amount	276,748						276,748	248,630
Repayment of secured financings
Total	$ 276,748						$ 276,748	248,630
Secured financing agreements | Woodstar Portfolio Mortgages | Minimum
Secured Financing Agreements
Interest rate (as a percent)	3.72%						3.72%
Secured financing agreements | Woodstar Portfolio Mortgages | Maximum
Secured Financing Agreements
Interest rate (as a percent)	3.97%						3.97%
Secured financing agreements | Woodstar Portfolio Government Financing
Secured Financing Agreements
Pledged Asset Carrying Value	$ 314,441						$ 314,441
Maximum Facility Size	135,584						135,584
Principal Amount	135,584						135,584	8,982
Repayment of secured financings
Total	135,584						$ 135,584	8,982
Secured financing agreements | Woodstar Portfolio Government Financing | Minimum
Secured Financing Agreements
Pricing margin (as a percent)							1.00%
Secured financing agreements | Woodstar Portfolio Government Financing | Maximum
Secured Financing Agreements
Pricing margin (as a percent)							5.00%
Secured financing agreements | Medical Office Portfolio Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value	767,540						$ 767,540
Maximum Facility Size	524,499						524,499
Principal Amount	491,197						491,197
Repayment of secured financings
Total	491,197						$ 491,197
Secured financing agreements | Medical Office Portfolio Mortgages | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.50%
Secured financing agreements | Term Loan A
Secured Financing Agreements
Pledged Asset Carrying Value	1,095,189						$ 1,095,189
Maximum Facility Size	300,000						300,000
Principal Amount	300,000						300,000
Repayment of secured financings
Total	300,000						$ 300,000
Secured financing agreements | Term Loan A | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.25%
Secured financing agreements | Term Loan B
Secured Financing Agreements
Principal Amount								658,270
Repayment of secured financings
Total								658,270
Secured financing agreements | Revolving Secured Financing
Secured Financing Agreements
Maximum Facility Size	100,000						$ 100,000
Secured financing agreements | Revolving Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)							2.25%
Secured financing agreements | FHLB Advances
Secured Financing Agreements
Principal Amount								9,250
Repayment of secured financings
Total								9,250
Repurchase Agreements
Secured Financing Agreements
Principal Amount	2,476,277						$ 2,476,277	2,655,264
Repayment of secured financings
2017	655,791						655,791
2018	622,858						622,858
2019	593,184						593,184
2020	439,752						439,752
2021	81,456						81,456
Thereafter	83,236						83,236
Total	2,476,277						2,476,277	2,655,264
Repurchase Agreements | Loans held for investment
Secured Financing Agreements
Principal Amount	1,890,925						1,890,925	2,142,198
Repayment of secured financings
Total	1,890,925						1,890,925	2,142,198
Repurchase Agreements | Loans held-for-sale
Secured Financing Agreements
Principal Amount	34,024						34,024	146,782
Repayment of secured financings
Total	34,024						34,024	146,782
Repurchase Agreements | Investment securities.
Secured Financing Agreements
Principal Amount	551,328						551,328	366,284
Repayment of secured financings
Total	551,328						551,328	$ 366,284
Other Secured Financing
Secured Financing Agreements
Principal Amount	1,720,941						1,720,941
Repayment of secured financings
2017	25,689						25,689
2018	53,889						53,889
2019	39,726						39,726
2020	322,578						322,578
2021	315,575						315,575
Thereafter	963,484						963,484
Total	$ 1,720,941						$ 1,720,941