Consolidated Statements of Income - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Interest Income
Residential loans	$ 315,953	$ 239,818	$ 154,362
Business purpose residential loans	53,805	4,333	0
Multifamily loans	132,600	21,322	0
Real estate securities	91,822	105,078	90,803
Other interest income	28,101	8,166	2,892
Total interest income	622,281	378,717	248,057
Interest Expense
Short-term debt	(96,506)	(58,917)	(36,851)
Asset-backed securities issued	(294,466)	(99,429)	(19,108)
Long-term debt	(88,836)	(80,693)	(52,857)
Total interest expense	(479,808)	(239,039)	(108,816)
Net Interest Income	142,473	139,678	139,241
Non-interest Income
Mortgage banking activities, net	87,266	59,566	53,908
Investment fair value changes, net	35,500	(25,689)	10,374
Other income	19,257	13,070	12,436
Realized gains, net	23,821	27,041	13,355
Total non-interest income, net	165,844	73,988	90,073
General and administrative expenses	(118,672)	(82,782)	(77,156)
Other expenses	13,022	196	0
Net Income before Provision for Income Taxes	176,623	130,688	152,158
Provision for income taxes	(7,440)	(11,088)	(11,752)
Net Income	$ 169,183	$ 119,600	$ 140,406
Basic earnings per common share (in dollars per share)	$ 1.63	$ 1.47	$ 1.78
Diluted earnings per common share (in dollars per share)	$ 1.46	$ 1.34	$ 1.60
Basic weighted average shares outstanding (in shares)	101,120,744	78,724,912	76,792,957
Diluted weighted average shares outstanding (in shares)	136,780,594	110,027,770	101,975,008