Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value of the Company's Financial Assets (Details) - USD ($) $ in Thousands		Jun. 30, 2016	Dec. 31, 2015		Jun. 30, 2015	Dec. 31, 2014
Financial Assets:
Cash and cash equivalents		$ 49,941	$ 61,959		$ 106,461	$ 75,598
Investment securities available for sale	[1]	796,489	765,454
Derivative assets, carrying value		291,680	228,775
Mezzanine loan and equity investments, carrying value		144,432	83,995
Financial Liabilities:
Outstanding Financing Arrangements		618,050	577,413	[2]
Derivative liabilities		6,438	1,500
Subordinated debentures, carrying value		45,000	45,000
Available-for-sale securities
Financial Liabilities:
Assets held in securitization trusts		42,271	40,734
Portfolio Investments
Financial Liabilities:
Outstanding Financing Arrangements		618,050	577,413
Residential collateralized debt obligations
Financial Liabilities:
Secured debt		102,597	116,710
Multi-family collateralized debt obligations
Financial Liabilities:
Secured debt		6,981,813	6,818,901
Level 1
Financial Assets:
Cash and cash equivalents		49,941	61,959
Cash and cash equivalents, estimated fair value		49,941	61,959
Financial Liabilities:
Payable for securities purchased, carrying value		286,452	227,969
Payable for securities purchased, estimated fair value		286,452	227,969
Level 2 | Portfolio Investments
Financial Liabilities:
Outstanding Financing Arrangements		618,050	577,413
Level 2 | Distressed Residential Mortgage Loans
Financial Liabilities:
Outstanding Financing Arrangements		174,798	212,155
Level 3
Financial Assets:
Mortgage loans held for sale (net), carrying value	[3]	5,283	5,471
Mortgage loans held for sale (net), estimated fair value	[3]	5,328	5,557
First mortgage loans, carrying value	[3]	10,391	2,706
First mortgage loans, estimated fair value	[3]	10,531	2,846
Mezzanine loan and equity investments, carrying value	[4]	75,300	44,151
Mezzanine loan and equity investments, estimated fair value	[4]	76,004	44,540
Equity method investment	[5]	73,839	87,662
Investments in unconsolidated entities, estimated fair value	[5]	74,304	87,558
Financial Liabilities:
Secured debt		244,016	116,541
Secured debt, estimated fair value		252,252	123,776
Subordinated debentures, carrying value		45,000	45,000
Subordinated debentures, estimated fair value		44,081	42,731
Level 3 | Residential mortgage loans held in securitization trusts
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value		106,173	119,921
Multi-family loans held in securitization trusts, estimated fair value		93,314	109,120
Level 3 | Distressed Residential Mortgage Loans
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value	[6]	543,361	558,989
Multi-family loans held in securitization trusts, estimated fair value	[6]	544,858	564,310
Level 3 | Multi-family loans held in securitization trusts
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value		7,282,145	7,105,336
Multi-family loans held in securitization trusts, estimated fair value		7,282,145	7,105,336
Level 3 | Residential collateralized debt obligations
Financial Liabilities:
Secured debt		102,597	116,710
Secured debt, estimated fair value		92,522	105,606
Level 3 | Multi-family collateralized debt obligations
Financial Liabilities:
Secured debt		6,981,813	6,818,901
Secured debt, estimated fair value		6,981,813	6,818,901
Level 1 and 2
Financial Assets:
Derivative assets, carrying value		291,680	228,775
Financial Liabilities:
Derivative liabilities		6,438	1,500
Distressed Residential Mortgage Loans
Financial Assets:
Residential mortgage loans held in securitization trusts (net), carrying value		$ 318,000	$ 444,800

[1] Includes $42.3 million and $40.7 million of investment securities for sale held in securitization trusts as of June 30, 2016 and December 31, 2015, respectively.

[2] Includes FHLBI advances amounting to $121.0 million as of December 31, 2015.

[3] Included in receivables and other assets in the accompanying condensed consolidated balance sheets.

[4] Includes mezzanine loan and preferred equity investments accounted for as loans (see Note 2).

[5] Includes investments in unconsolidated entities accounted for under the fair value option with a carrying value of $63.1 million and $67.6 million at June 30, 2016 and December 31, 2015, respectively.

[6] Includes distressed residential mortgage loans held in securitization trusts with a carrying value amounting to approximately $225.4 million and $114.2 million at June 30, 2016 and December 31, 2015, respectively, and distressed residential mortgage loans with a carrying value amounting to approximately $318.0 million and $444.8 million at June 30, 2016 and December 31, 2015, respectively.