Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value of the Company's Financial Assets (Details) - USD ($) $ in Thousands		Dec. 31, 2015		Dec. 26, 2015	Dec. 31, 2014		Dec. 31, 2013	Dec. 31, 2012
Financial Assets:
Cash and cash equivalents		$ 61,959			$ 75,598		$ 31,798	$ 31,777
Investment securities available for sale		724,720			816,647
Investment securities, available for sale, at fair value held in securitization trusts		40,734	[1]		38,594	[2]
Derivative assets		228,775			288,850
Receivables and other assets		219,130			222,491
Financial Liabilities:
Outstanding Borrowings		577,413			651,965
Derivative liabilities		1,500			1,463
Subordinated debentures		45,000			45,000
Financing arrangements, portfolio investments
Financial Liabilities:
Outstanding Borrowings		577,413			651,965
Distressed Residential Mortgage Loans
Financial Liabilities:
Outstanding Borrowings		214,490			238,949
Residential collateralized debt obligations
Financial Liabilities:
Collateralized debt obligations		116,710			145,542
Multi-family collateralized debt obligations, at fair value
Financial Liabilities:
Collateralized debt obligations		6,818,901			8,048,053
Distressed residential mortgage loans held in securitization trusts (net)
Financial Assets:
Residential mortgage loans		114,214	[3]	$ 5,500	221,591
Financial Liabilities:
Carrying value of distressed residential mortgage loans held in securitization trusts		114,200			221,600
Distressed Residential Mortgage Loans
Financial Assets:
Residential mortgage loans		444,800			361,100
Financial Liabilities:
Carrying value of distressed residential mortgage loans held in securitization trusts		559,000			582,700
Level 1 or 2
Financial Assets:
Investment securities available for sale		724,720			816,647
Derivative assets		228,775			288,850
Financial Liabilities:
Derivative liabilities		1,500			1,463
Level 1
Financial Assets:
Cash and cash equivalents		61,959			75,598
Cash and cash equivalents, fair value		61,959			75,598
Financial Liabilities:
Payables for securities purchased		227,969			283,537
Payables for securities purchased, fair value		227,969			283,537
Level 2 | Financing arrangements, portfolio investments
Financial Liabilities:
Outstanding Borrowings		577,413			651,965
Level 2 | Distressed Residential Mortgage Loans
Financial Liabilities:
Outstanding Borrowings		214,490			238,949
Level 3
Financial Assets:
Investment securities, available for sale, at fair value held in securitization trusts		40,734			38,594
Mortgage loans held for sale (net)	[4]	5,471			7,712
Mortgage loans held for sale, fair value	[4]	5,557			7,713
First mortgage loans	[4]	2,706			1,760
First mortgage loans, fair value	[4]	2,846			1,900
Receivables and other assets	[4],[5]	44,151			24,907
Receivables and other assets, fair value	[4],[5]	44,540			25,212
Investment in unconsolidated entities	[6]	87,065			48,366
Investments in unconsolidated entities, fair value	[6]	87,558			48,490
Financial Liabilities:
Collateralized debt obligations		117,528			232,877
Collateralized debt obligations, fair value		123,776			240,341
Subordinated debentures		45,000			45,000
Subordinated debentures, fair value		42,731			36,531
Level 3 | Residential collateralized debt obligations
Financial Liabilities:
Collateralized debt obligations		116,710			145,542
Collateralized debt obligations, fair value		105,606			130,919
Level 3 | Multi-family collateralized debt obligations, at fair value
Financial Liabilities:
Collateralized debt obligations		6,818,901			8,048,053
Collateralized debt obligations, fair value		6,818,901			8,048,053
Level 3 | Residential mortgage loans held in securitization trusts
Financial Assets:
Residential mortgage loans		119,921			149,614
Multi-family loans held in securitization trusts		109,120			135,241
Level 3 | Distressed Residential Mortgage Loans
Financial Assets:
Residential mortgage loans	[7]	558,989			582,697
Multi-family loans held in securitization trusts	[7]	564,310			599,182
Level 3 | Multi Family Loans Held In Securitization Trusts Net
Financial Assets:
Residential mortgage loans		7,105,336			8,365,514
Multi-family loans held in securitization trusts		7,105,336			8,365,514
Equity Method Investments
Financial Assets:
Investment in unconsolidated entities		67,600			38,500
Fair Value, Measurements, Recurring
Financial Assets:
Investments in unconsolidated entities, fair value		67,571			38,496
Fair Value, Measurements, Recurring | Level 3
Financial Assets:
Investments in unconsolidated entities, fair value		67,571			38,496
U.S. Treasury securities
Financial Assets:
Investment securities available for sale		$ 10,037			$ 0

[1] The Company classified the multi-family CMBS issued by two K-Series securitizations and held by this Financing VIE as available for sale securities as the purpose is not to trade these securities. The Financing VIE consolidated one K-Series securitization that issued certain of the multi-family CMBS owned by the Company, including its assets, liabilities, income and expenses, in its financial statements, as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in this particular K-Series securitization (see Note 6).

[2] The Company classified the multi-family CMBS issued by two K-Series securitizations and held by the Financing VIE as available for sale securities as the purpose is not to trade these securities. The Financing VIE consolidated one K-Series securitization that issued certain of the multi-family CMBS owned by the Company, including its assets, liabilities, income and expenses, in its financial statements, as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in this particular K-Series securitization (see Note 6).

[3] In December 2015, the Company repaid the Company’s outstanding notes from its distressed residential mortgage loan securitization transaction completed in December 2012 with an original principal amount of $38.7 million and outstanding principal balance at the time of repayment amounting to $5.5 million. With the repayment of the notes, the Company terminated and deconsolidated the Financing VIE that facilitated this financing transaction and the distressed residential loans serving as collateral on the notes were transferred back to the Company.

[4] Included in receivables and other assets in the accompanying consolidated balance sheets.

[5] Includes mezzanine and preferred equity investments accounted for as loans (see Note 2).

[6] Includes investments in unconsolidated entities accounted for under the fair value option with a carrying value of $67.6 million and $38.5 million at December 31, 2015 and December 31, 2014, respectively.

[7] Includes distressed residential mortgage loans held in securitization trusts with a carrying value amounting to approximately $114.2 million and $221.6 million at December 31, 2015 and December 31, 2014, respectively and distressed residential mortgage loans with a carrying value amounting to approximately $444.8 million and $361.1 million at December 31, 2015 and December 31, 2014, respectively.