Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value of the Companys Financial Assets (Details) - USD ($) $ in Thousands		Mar. 31, 2016	Dec. 31, 2015	Mar. 31, 2015	Dec. 31, 2014
Financial Assets:
Cash and cash equivalents, carrying value		$ 39,931	$ 61,959	$ 88,990	$ 75,598
Investment securities available for sale		794,473	765,454
Derivative assets, carrying value		288,925	228,775
Mezzanine loan and equity investments, carrying value		226,369	215,808
Receivable for securities sold, carrying value		1,858	0
Financial Liabilities:
Outstanding Financing Arrangements		589,919	577,413
Derivative liabilities		4,998	1,500
Subordinated debentures, carrying value		45,000	45,000
Portfolio Investments
Financial Liabilities:
Outstanding Financing Arrangements		589,919	577,413
Distressed Residential Mortgage Loans
Financial Liabilities:
Outstanding Financing Arrangements		216,604	212,155
Residential collateralized debt obligations
Financial Liabilities:
Secured debt		110,023	116,710
Multi-family collateralized debt obligations
Financial Liabilities:
Secured debt		6,957,293	6,818,901
Level 1
Financial Assets:
Cash and cash equivalents, carrying value		39,931	61,959
Cash and cash equivalents, estimated fair value		39,931	61,959
Receivable for securities sold, carrying value		1,858
Financial Liabilities:
Payable for securities purchased, carrying value		311,250	227,969
Payable for securities purchased, estimated fair value		311,250	227,969
Level 2 | Portfolio Investments
Financial Liabilities:
Outstanding Financing Arrangements		589,919	577,413
Level 2 | Distressed Residential Mortgage Loans
Financial Liabilities:
Outstanding Financing Arrangements		216,604	212,155
Level 3
Financial Assets:
Mortgage loans held for sale (net), carrying value	[1]	5,755	5,471
Mortgage loans held for sale (net), estimated fair value	[1]	5,859	5,557
First mortgage loans, carrying value	[1]	6,501	2,706
First mortgage loans, estimated fair value	[1]	6,641	2,846
Mezzanine loan and equity investments, carrying value	[1],[2]	44,355	44,151
Mezzanine loan and equity investments, estimated fair value	[1],[2]	44,717	44,540
Equity method investment	[3]	85,497	87,065
Investments in unconsolidated entities, estimated fair value	[3]	86,090	87,558
Financial Liabilities:
Secured debt		83,471	116,541
Secured debt, estimated fair value		89,742	123,776
Subordinated debentures, carrying value		45,000	45,000
Subordinated debentures, estimated fair value		33,846	42,731
Level 3 | Residential mortgage loans held in securitization trusts
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value		113,186	119,921
Multi-family loans held in securitization trusts, estimated fair value		100,909	109,120
Level 3 | Distressed Residential Mortgage Loans
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value	[4]		558,989
Multi-family loans held in securitization trusts, estimated fair value	[4]	547,818	564,310
Level 3 | Multi-family loans held in securitization trusts
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value		7,250,586	7,105,336
Multi-family loans held in securitization trusts, estimated fair value		7,250,586	7,105,336
Level 3 | Residential collateralized debt obligations
Financial Liabilities:
Secured debt		110,023	116,710
Secured debt, estimated fair value		98,113	105,606
Level 3 | Multi-family collateralized debt obligations
Financial Liabilities:
Secured debt		6,957,293	6,818,901
Secured debt, estimated fair value		6,957,293	6,818,901
Level 1 or 2
Financial Assets:
Derivative assets, carrying value		288,925	228,775
Financial Liabilities:
Derivative liabilities		4,998	1,500
Level 1, 2 or 3
Financial Assets:
Investment securities available for sale	[5]	$ 794,473	$ 765,454

[1] Included in receivables and other assets in the accompanying condensed consolidated balance sheets.

[2] Includes mezzanine and preferred equity investments accounted for as loans (see Note 2).

[3] Includes investments in unconsolidated entities accounted for under the fair value option with a carrying value of $65.8 million and $67.6 million at March 31, 2016 and December 31, 2015, respectively.

[4] Includes distressed residential mortgage loans held in securitization trusts with a carrying value amounting to approximately $0 and $114.2 million at March 31, 2016 and December 31, 2015, respectively, and distressed residential mortgage loans with a carrying value amounting to approximately $537.6 million and $444.8 million at March 31, 2016 and December 31, 2015, respectively.

[5] Includes $41.5 million and $40.7 million of investment securities for sale held in securitization trusts as of March 31, 2016 and December 31, 2015, respectively.