Quarterly Financial Data (unaudited) (Details) - USD ($) $ / shares in Units, shares in Thousands, $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014	Sep. 30, 2014	Jun. 30, 2014	Mar. 31, 2014	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 80,866	$ 82,587	$ 84,400	$ 88,985	$ 93,124	$ 94,937	$ 95,548	$ 95,238	$ 336,838	$ 378,847	$ 291,727
Interest expense	64,875	64,295	64,097	67,384	74,315	75,617	75,665	75,413	13,737	5,569	6,655
NET INTEREST INCOME	15,991	18,292	20,303	21,601	18,809	19,320	19,883	19,825	76,187	77,837	60,549
Other Income (loss):
Recovery (provision) for loan losses	302	(1,117)	(112)	(436)	(705)	(82)	(663)	(489)	(1,363)	(1,939)	(1,262)
Realized gain(loss) on investment securities and related hedges, net	(1,555)	(2,895)	(1,291)	1,124	21,672	17,055	1,325	2,039	(4,617)	42,091	(10,719)
Gain on de-consolidation of multi-family loans held in securitization trust and multi-family collateralized debt obligations	0	0	0	1,483					1,483	0	0
Realized gain(loss) on distressed residential mortgage loans	(263)	27,224	3,614	676	4,903	834	418	8,225
Unrealized (loss) gain on investment securities and related hedges, net	1,002	(2,631)	4,716	(5,728)	(3,620)	(1,020)	(1,291)	(1,736)	(2,641)	(7,667)	5,464
Unrealized gain(loss) on multi-family loans and debt held in securitization trusts, net	(4,508)	(2,170)	5,418	13,628	13,871	18,115	20,019	4,926	12,368	56,931	31,495
Loss on extinguishment of debt					0	(3,397)	0	0	0	(3,397)	0
Other income	2,967	1,807	2,300	2,286	2,483	1,613	203	510
Total other income (loss)	(2,055)	20,218	14,645	13,033	38,604	33,118	20,011	13,475	45,841	105,208	29,062
General, administrative and other expenses	9,665	9,830	9,139	10,846	13,710	11,613	7,577	7,559	9,928	9,500	7,916
Income from operations before income taxes	4,271	28,680	25,809	23,788	43,703	40,825	32,317	25,741
Income tax expense	64	3,048	1,178	245	1,727	1,100	538	3,030	4,535	6,395	739
NET INCOME	4,207	25,632	24,631	23,543	41,976	39,725	31,779	22,711	78,013	136,191	68,955
Preferred Stock Dividends	(3,225)	(3,225)	(3,087)	(1,453)	(1,453)	(1,453)	(1,453)	(1,453)	(10,990)	(5,812)	(3,568)
NET INCOME ATTRIBUTABLE TO COMMON STOCKHOLDERS	$ 982	$ 22,407	$ 21,544	$ 22,090	$ 40,523	$ 38,272	$ 30,326	$ 21,258	$ 67,023	$ 130,379	$ 65,387
Per share basic income (in dollars per share)	$ 0.01	$ 0.20	$ 0.20	$ 0.21	$ 0.42	$ 0.42	$ 0.34	$ 0.29	$ 0.62	$ 1.48	$ 1.11
Per share diluted income (in dollars per share)	0.01	0.20	0.20	0.21	0.42	0.42	0.34	0.29	$ 0.62	$ 1.48	$ 1.11
Dividends declared per common share (in dollars per share)	$ 0.24	$ 0.24	$ 0.27	$ 0.27	$ 0.27	$ 0.27	$ 0.27	$ 0.27
Weighted average shares outstanding-basic (in shares)	109,402	109,402	109,252	105,488	96,323	90,685	89,686	74,505	108,399	87,867	59,102
Weighted average shares outstanding-diluted (in shares)	109,402	109,402	109,252	105,488	96,323	90,685	89,686	74,505	108,399	87,867	59,102