Condensed Consolidated Statements of Operations - USD ($) shares in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015
INTEREST INCOME:
Investment securities and other	$ 8,591,000	$ 10,196,000	$ 17,025,000	$ 21,540,000
Total interest income	79,766,000	84,400,000	161,392,000	173,385,000
INTEREST EXPENSE:
Interest expense	3,962,000	3,442,000	7,811,000	6,905,000
Subordinated debentures	508,000	468,000	1,009,000	928,000
Total interest expense	63,102,000	64,097,000	127,086,000	131,481,000
NET INTEREST INCOME	16,664,000	20,303,000	34,306,000	41,904,000
OTHER INCOME (LOSS):
Recovery (provision) for loan losses	42,000	(112,000)	688,000	(548,000)
Realized gain (loss) on investment securities and related hedges, net	1,761,000	(1,291,000)	3,027,000	(167,000)
Gain on de-consolidation of multi-family loans held in securitization trust and multi-family collateralized debt obligations	0	0	0	1,483,000
Realized gain on distressed residential mortgage loans	26,000	3,614,000	5,574,000	4,290,000
Unrealized (loss) gain on investment securities and related hedges, net	(667,000)	4,716,000	(3,159,000)	(1,012,000)
Unrealized gain on multi-family loans and debt held in securitization trusts, net	784,000	5,418,000	1,602,000	19,046,000
Other income	8,125,000	2,300,000	11,198,000	4,586,000
Total other income	10,071,000	14,645,000	18,930,000	27,678,000
Base management and incentive fees	2,979,000	4,141,000	6,504,000	11,011,000
Expenses related to distressed residential mortgage loans	2,740,000	2,682,000	5,934,000	4,566,000
Other general and administrative expenses	4,217,000	2,316,000	6,857,000	4,408,000
Total general, administrative and other expenses	9,936,000	9,139,000	19,295,000	19,985,000
INCOME FROM OPERATIONS BEFORE INCOME TAXES	16,799,000	25,809,000	33,941,000	49,597,000
Income tax expense	2,366,000	1,178,000	2,557,000	1,423,000
NET INCOME	14,433,000	24,631,000	31,384,000	48,174,000
Net loss attributable to non-controlling interest	2,000	0	1,711	0
NET INCOME ATTRIBUTABLE TO COMPANY	14,435,000	24,631,000	31,386,000	48,174,000
Preferred stock dividends	(3,225,000)	(3,087,000)	(6,450,000)	(4,540,000)
NET INCOME ATTRIBUTABLE TO COMPANY'S COMMON STOCKHOLDERS	$ 11,210,000	$ 21,544,000	$ 24,936,000	$ 43,634,000
Basic income per common share (in Dollars per share)	$ 0.10	$ 0.20	$ 0.23	$ 0.41
Diluted income per common share (in Dollars per share)	$ 0.10	$ 0.20	$ 0.23	$ 0.41
Weighted average shares outstanding-basic (in Shares)	109,489	109,252	109,445	107,380
Weighted average shares outstanding-diluted (in Shares)	109,489	109,252	109,445	107,380
Securitized debt
INTEREST EXPENSE:
Interest expense	$ 3,096,000	$ 2,974,000	$ 5,227,000	$ 6,101,000
Multi-family loans held in securitization trusts
INTEREST INCOME:
Interest income	61,769,000	62,984,000	125,301,000	129,284,000
Residential mortgage loans held in securitization trusts
INTEREST INCOME:
Interest income	921,000	895,000	1,757,000	2,075,000
Distressed residential mortgage loans
INTEREST INCOME:
Interest income	8,485,000	10,325,000	17,309,000	20,486,000
Multi-family collateralized debt obligations
INTEREST EXPENSE:
Interest expense	55,224,000	56,992,000	112,424,000	117,087,000
Residential collateralized debt obligations
INTEREST EXPENSE:
Interest expense	$ 312,000	$ 221,000	$ 615,000	$ 460,000