Quarterly Results of Operations (Details) - USD ($) $ / shares in Units, $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014	Sep. 30, 2014	Jun. 30, 2014	Mar. 31, 2014	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Interest income:
Securities	$ 72,211	$ 70,874	$ 76,270	$ 86,362	$ 87,702	$ 80,969	$ 89,458	$ 96,307	$ 305,717	$ 354,436	$ 450,708
Mortgage loans	2,168	1,786	1,088	467	35
Mortgage loans in securitization trusts	446								446
Short-term cash investments	433	328	318	288	324	330	347	282	1,367	1,283	1,560
Total interest income	75,258	72,988	77,676	87,117	88,061	81,299	89,805	96,589
Interest expense:
Repurchase agreements	18,724	20,956	23,750	27,314	26,966	31,950	35,128	38,451	90,744	132,495	197,709
Warehouse lines	305	292							597
Collateralized borrowings in securitization trusts	97								97
Total interest expense	19,126	21,248	23,750	27,314	26,966	31,950	35,128	38,451	91,438	132,495	197,709
Net interest margin	56,132	51,740	53,926	59,803	61,095	49,349	54,677	58,138	221,601	223,259	254,559
Other income (loss):
Servicing income	15,549	4,562
Management fee income	1,503	524							2,027
Net realized gain on sale of securities	15,371	1,216	(309)	16,453	2,107	237	(4,584)	7,436
Net gain (loss) on mortgage loans	(1,950)	1,361	(689)	244	8				(1,034)	8
Net gain on mortgage servicing rights	11,250	(539)							10,711
Net gain (loss) on derivative instruments	64,085	(130,301)	(19,415)	(102,785)	(79,988)	35,430	(55,260)	(41,615)	(188,416)	(141,433)	(69,715)
Net miscellaneous gains and losses	1,686	(912)							774
Total other income (loss)	107,494	(124,089)	(20,413)	(86,088)	(77,873)	35,667	(59,844)	(34,179)	(123,096)	(136,229)	(360,829)
Expenses:
Management fee	4,020	4,037	4,083	4,095	4,112	4,122	4,144	4,154
Share-based compensation	1,211	966	1,034	1,045	1,020	890	842	860	4,256	3,612	2,594
Servicing expenses	2,268	754
General and administrative	9,163	5,786	3,840	3,110	3,941	2,113	2,324	2,147	21,899	10,525	7,092
Securitization deal costs	1,473								1,473
Total expenses	18,135	11,543	8,957	8,250	9,073	7,125	7,310	7,161
Net income	145,491	(83,892)	24,556	(34,535)	(25,851)	77,891	(12,477)	16,798	51,620	56,361	(134,136)
Dividends on preferred stock	5,481	5,480	5,480	5,481	5,481	5,480	5,481	5,480
Net income (loss) available to common shareholders	$ 140,010	$ (89,372)	$ 19,076	$ (40,016)	$ (31,332)	$ 72,411	$ (17,958)	$ 11,318	$ 29,698	$ 34,439	$ (156,058)
Earnings (loss) per share - common stock, basic	$ 1.45	$ (0.93)	$ 0.20	$ (0.41)	$ (0.32)	$ 0.75	$ (0.19)	$ 0.12	$ 0.31	$ 0.36	$ (1.59)
Earnings (loss) per share - common stock, diluted	1.45	(0.93)	0.20	(0.41)	(0.32)	0.75	(0.19)	0.12	0.31	0.36	(1.59)
Dividends per share of common stock	$ 0.45	$ 0.45	$ 0.50	$ 0.50	$ 0.50	$ 0.50	$ 0.50	$ 0.50	$ 1.90	$ 2.00	$ 2.45
Weighted average common shares outstanding, basic	96,546,219	96,545,913	96,790,541	96,783,199	96,728,821	96,563,132	96,515,599	96,606,081	96,665,489	96,603,634	98,337,362
Weighted average common shares outstanding, diluted	96,546,219	96,545,913	96,790,541	96,783,199	96,728,821	96,563,132	96,515,599	96,606,081	96,665,489	96,603,634	98,337,362