Summary of Quarterly Results of Operations (Unaudited) (Details) - USD ($) $ / shares in Units, $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 112,528	$ 112,716	$ 114,507	$ 117,418	$ 118,499	$ 119,706	$ 123,995	$ 129,943	$ 457,169	$ 492,143	$ 463,817
Interest expense	(49,868)	(48,167)	(47,720)	(47,600)	(46,456)	(43,703)	(42,849)	(43,940)	(193,355)	(176,948)	(159,808)
Net Interest Income	62,660	64,549	66,787	69,818	72,043	76,003	81,146	86,003	263,814	315,195	304,009
Net Impairment Losses Recognized in Earnings	0	(485)	0	0	0	0	(298)	(407)	(485)	(705)	0
Net gain on residential whole loans held at fair value	14,632	18,701	14,470	11,881	6,899	5,565	3,224	2,034	59,684	17,722	116
Gain on sales of MBS	9,768	7,083	9,241	9,745	9,652	11,196	7,617	6,435	35,837	34,900	37,497
Other income	1,281	8,117	3,319	1,085	(831)	(259)	(678)	311
Operating and other expense	(15,704)	(14,954)	(14,867)	(14,459)	(14,292)	(12,995)	(12,940)	(12,202)	(59,984)	(52,429)	(45,290)
Net Income	72,637	83,011	78,950	78,070	73,471	79,510	78,071	82,174	312,668	313,226	313,504
Preferred stock dividends	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(15,000)	(15,000)	(15,000)
Net Income Available to Common Stock and Participating Securities	$ 68,887	$ 79,261	$ 75,200	$ 74,320	$ 69,721	$ 75,760	$ 74,321	$ 78,424	$ 297,668	$ 298,226	$ 298,504
Earnings per Common Share - Basic and Diluted (in dollars per share)	$ 0.18	$ 0.21	$ 0.20	$ 0.20	$ 0.19	$ 0.20	$ 0.20	$ 0.21	$ 0.80	$ 0.80	$ 0.81