Secured Financing Agreements (Details) $ in Thousands	3 Months Ended		12 Months Ended
	Mar. 31, 2016 USD ($) item	Mar. 31, 2015 USD ($)	Dec. 31, 2015 USD ($)	Feb. 29, 2016 USD ($)
Secured Financing Agreements
Unamortized premium, net	$ 1,099
Unamortized discount			$ 1,702
Unamortized deferred financing costs	(36,147)		(38,336)
Carrying Value	$ 4,480,960		$ 3,980,699
Repayment of secured financings
Percentage of repurchase agreements for which margin calls are limited to collateral specific credit marks	59.00%
Percentage of repurchase agreements containing margin call provisions that pertain to loans held-for-sale	27.00%
Lender 2 Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size	$ 600,000			$ 500,000
Amount of increase in available borrowings	100,000
Lender 4 Repo 2 Facility
Secured Financing Agreements
Maximum borrowing capacity	600,000
Maximum facility size subject to certain conditions	1,000,000
Lender 7 Secured Financing
Secured Financing Agreements
Maximum borrowing capacity	450,000
Maximum facility size subject to certain conditions	$ 650,000
Floor interest rate (as a percent)	75.00%
CMBS Repo 1 Facility
Secured Financing Agreements
Maturity period	11 months
CMBS Repo 3 Facility
Secured Financing Agreements
Maturity period	12 months
RMBS Repo 1 Facility
Secured Financing Agreements
Current maturity period, relative to when the buyer delivers notice to the seller	180 days		180 days
Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Interest rate (as a percent)	3.00%
Maximum Facility Size	$ 16,000
Number of investment securities financed | item	2
Investing and Servicing Segment Property Mortgages | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.25%
Investing and Servicing Segment Property Mortgages | Weighted-average
Secured Financing Agreements
Maturity period	4 years 10 months 24 days
Woodstar Portfolio Government Financing
Secured Financing Agreements
Principal Amount	$ 126,700
Number of federal, state and county sponsored mortgage facilities assumed | item	16
Repayment of secured financings
Total	$ 126,700
Secured financing agreements
Secured Financing Agreements
Pledged Asset Carrying Value	9,088,287
Maximum Facility Size	7,209,205
Principal Amount	4,516,008		$ 4,020,737
Repayment of secured financings
2016 (remainder of)	370,204
2017	1,196,441
2018	671,362
2019	613,486
2020	1,153,608
Thereafter	510,907
Total	4,516,008		4,020,737
Additional non-cash interest expense	3,900	$ 3,500
Secured financing agreements | Lender 1 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	2,155,043
Maximum Facility Size	1,600,000
Principal Amount	1,340,634		975,735
Repayment of secured financings
Total	$ 1,340,634		975,735
Secured financing agreements | Lender 1 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.85%
Secured financing agreements | Lender 1 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	5.25%
Secured financing agreements | Lender 2 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 285,237
Maximum Facility Size	500,000
Principal Amount	175,912		233,705
Repayment of secured financings
Total	$ 175,912		233,705
Secured financing agreements | Lender 2 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.75%
Secured financing agreements | Lender 2 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.75%
Secured financing agreements | Lender 3 Repo I Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 152,414
Maximum Facility Size	109,745
Principal Amount	109,745		131,997
Repayment of secured financings
Total	$ 109,745		131,997
Secured financing agreements | Lender 3 Repo I Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.50%
Secured financing agreements | Lender 3 Repo I Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.85%
Secured financing agreements | Lender 4 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 385,003
Maximum Facility Size	301,518
Principal Amount	301,518		309,498
Repayment of secured financings
Total	$ 301,518		309,498
Secured financing agreements | Lender 4 Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.00%
Secured financing agreements | Lender 4 Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 192,257
Maximum Facility Size	1,000,000
Principal Amount	151,999
Repayment of secured financings
Total	$ 151,999
Secured financing agreements | Lender 4 Repo 2 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.50%
Secured financing agreements | Lender 6 Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 702,162
Maximum Facility Size	500,000
Principal Amount	413,104		491,263
Repayment of secured financings
2016 (remainder of)	221,900
Total	$ 413,104		491,263
Secured financing agreements | Lender 6 Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.50%
Secured financing agreements | Lender 6 Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	3.00%
Secured financing agreements | Lender 7 Secured Financing
Secured Financing Agreements
Pledged Asset Carrying Value	$ 109,285
Maximum Facility Size	$ 650,000
Principal Amount			38,055
Repayment of secured financings
Total			38,055
Secured financing agreements | Lender 7 Secured Financing | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.75%
Secured financing agreements | Conduit Repo 1 Facility
Secured Financing Agreements
Maximum Facility Size	$ 150,000
Principal Amount			80,741
Repayment of secured financings
Total			80,741
Secured financing agreements | Conduit Repo 1 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.95%
Secured financing agreements | Conduit Repo 1 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	3.35%
Secured financing agreements | Conduit Repo 2 Facility
Secured Financing Agreements
Maximum Facility Size	$ 150,000
Secured financing agreements | Conduit Repo 2 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.10%
Secured financing agreements | Conduit Repo 3 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 96,986
Maximum Facility Size	150,000
Principal Amount	71,599		66,041
Repayment of secured financings
2016 (remainder of)	71,600
Total	$ 71,599		66,041
Secured financing agreements | Conduit Repo 3 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.10%
Secured financing agreements | Conduit Repo 4 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 16,842
Maximum Facility Size	100,000
Principal Amount	12,188
Repayment of secured financings
Total	$ 12,188
Secured financing agreements | Conduit Repo 4 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.25%
Secured financing agreements | CMBS Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 32,710
Maximum Facility Size	21,354
Principal Amount	21,354
Repayment of secured financings
Total	$ 21,354
Secured financing agreements | CMBS Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.90%
Secured financing agreements | CMBS Repo 2 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 132,740
Maximum Facility Size	100,238
Principal Amount	100,238		120,850
Repayment of secured financings
Total	$ 100,238		120,850
Secured financing agreements | CMBS Repo 2 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.35%
Secured financing agreements | CMBS Repo 2 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.70%
Secured financing agreements | CMBS Repo 3 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 365,199
Maximum Facility Size	260,777
Principal Amount	260,777		243,434
Repayment of secured financings
Total	$ 260,777		243,434
Secured financing agreements | CMBS Repo 3 Facility | Minimum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.40%
Secured financing agreements | CMBS Repo 3 Facility | Maximum | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.85%
Secured financing agreements | RMBS Repo 1 Facility
Secured Financing Agreements
Pledged Asset Carrying Value	$ 168,001
Maximum Facility Size	125,000
Principal Amount	71,707		2,000
Repayment of secured financings
Total	$ 71,707		2,000
Secured financing agreements | RMBS Repo 1 Facility | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.90%
Secured financing agreements | Investing and Servicing Segment Property Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value	$ 133,136
Maximum Facility Size	106,055
Principal Amount	100,715		82,964
Repayment of secured financings
Total	100,715		82,964
Secured financing agreements | Ireland Portfolio Mortgage
Secured Financing Agreements
Pledged Asset Carrying Value	500,086
Maximum Facility Size	334,623
Principal Amount	334,623		319,322
Repayment of secured financings
Total	$ 334,623		319,322
Secured financing agreements | Ireland Portfolio Mortgage | EURIBOR
Secured Financing Agreements
Pricing margin (as a percent)	1.69%
Secured financing agreements | Woodstar Portfolio Mortgages
Secured Financing Agreements
Pledged Asset Carrying Value	$ 338,281
Maximum Facility Size	248,630
Principal Amount	248,630		248,630
Repayment of secured financings
Total	$ 248,630		248,630
Secured financing agreements | Woodstar Portfolio Mortgages | Minimum
Secured Financing Agreements
Interest rate (as a percent)	3.72%
Secured financing agreements | Woodstar Portfolio Mortgages | Maximum
Secured Financing Agreements
Interest rate (as a percent)	3.81%
Secured financing agreements | Woodstar Portfolio Government Financing
Secured Financing Agreements
Pledged Asset Carrying Value	$ 296,321
Maximum Facility Size	135,437
Principal Amount	135,437		8,982
Repayment of secured financings
Total	$ 135,437		8,982
Secured financing agreements | Woodstar Portfolio Government Financing | Minimum
Secured Financing Agreements
Interest rate (as a percent)	1.00%
Secured financing agreements | Woodstar Portfolio Government Financing | Maximum
Secured Financing Agreements
Interest rate (as a percent)	5.00%
Secured financing agreements | Term Loan
Secured Financing Agreements
Pledged Asset Carrying Value	$ 3,015,838
Maximum Facility Size	656,578
Principal Amount	656,578		658,270
Repayment of secured financings
Total	$ 656,578		658,270
Secured financing agreements | Term Loan | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	2.75%
Secured financing agreements | FHLB Advances
Secured Financing Agreements
Pledged Asset Carrying Value	$ 10,746
Maximum Facility Size	9,250
Principal Amount	9,250		9,250
Repayment of secured financings
Total	$ 9,250		9,250
Secured financing agreements | FHLB Advances | LIBOR
Secured Financing Agreements
Pricing margin (as a percent)	0.37%
Repurchase Agreements
Secured Financing Agreements
Pledged Asset Carrying Value	$ 3,030,775		2,655,264
Principal Amount	3,030,775
Repayment of secured financings
2016 (remainder of)	353,820
2017	1,186,779
2018	641,989
2019	594,290
2020	174,303
Thereafter	79,594
Total	3,030,775
Repurchase Agreements | Loan held for investment
Secured Financing Agreements
Pledged Asset Carrying Value	2,492,912		2,142,198
Repurchase Agreements | Loans held-for-sale
Secured Financing Agreements
Pledged Asset Carrying Value	83,787		146,782
Repurchase Agreements | Investment securities.
Secured Financing Agreements
Pledged Asset Carrying Value	454,076		$ 366,284
Other Secured Financing
Secured Financing Agreements
Principal Amount	1,485,233
Repayment of secured financings
2016 (remainder of)	16,384
2017	9,662
2018	29,373
2019	19,196
2020	979,305
Thereafter	431,313
Total	$ 1,485,233