Consolidated Statements of Income - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Interest Income
Residential loans	$ 154,362	$ 137,804	$ 114,715
Commercial loans	345	30,496	46,933
Real estate securities	90,803	76,873	97,448
Other interest income	2,547	1,182	336
Total interest income	248,057	246,355	259,432
Interest Expense
Short-term debt	(36,851)	(22,287)	(30,572)
Asset-backed securities issued	(19,108)	(14,735)	(21,469)
Long-term debt	(52,857)	(51,506)	(43,842)
Total interest expense	(108,816)	(88,528)	(95,883)
Net Interest Income	139,241	157,827	163,549
Reversal of provision for loan losses	0	7,102	355
Net Interest Income after Provision	139,241	164,929	163,904
Non-interest Income
Mortgage banking activities, net	53,908	38,691	10,972
Mortgage servicing rights income (loss), net	7,860	14,353	(3,922)
Investment fair value changes, net	10,374	(28,574)	(21,357)
Other income	4,576	6,338	3,192
Realized gains, net	13,355	28,009	36,369
Total non-interest income, net	90,073	58,817	25,254
Operating expenses	(77,156)	(88,786)	(97,416)
Net Income before Provision for Income Taxes	152,158	134,960	91,742
(Provision for) benefit from income taxes	(11,752)	(3,708)	10,346
Net Income	$ 140,406	$ 131,252	$ 102,088
Basic earnings per common share (in dollars per share)	$ 1.78	$ 1.66	$ 1.20
Diluted earnings per common share (in dollars per share)	1.60	1.54	1.18
Regular dividends declared per common share (in dollars per share)	$ 1.12	$ 1.12	$ 1.12
Basic weighted average shares outstanding (in shares)	76,792,957	76,747,047	82,945,103
Diluted weighted average shares outstanding (in shares)	101,975,008	97,909,090	84,518,395