Consolidated Statements of Income (Loss) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Interest Income
Residential loans	$ 222,746	$ 315,953	$ 239,818
Business purpose loans	217,617	53,805	4,333
Multifamily loans	54,813	132,600	21,322
Real estate securities	49,605	91,822	105,078
Other interest income	27,135	28,101	8,166
Total interest income	571,916	622,281	378,717
Interest Expense
Short-term debt	(50,895)	(96,506)	(58,917)
Asset-backed securities issued	(299,708)	(294,466)	(99,429)
Long-term debt	(97,402)	(88,836)	(80,693)
Total interest expense	(448,005)	(479,808)	(239,039)
Net Interest Income	123,911	142,473	139,678
Non-interest (Loss) Income
Mortgage banking activities, net	78,472	87,266	59,566
Investment fair value changes, net	(588,438)	35,500	(25,689)
Other income	4,188	19,257	13,070
Realized gains, net	30,424	23,821	27,041
Total non-interest income, net	(475,354)	165,844	73,988
General and administrative expenses	(115,204)	(108,737)	(75,298)
Loan acquisition costs	(11,023)	(9,935)	(7,484)
Other expenses	108,785	13,022	196
Net (Loss) Income before Benefit from (Provision for) Income Taxes	(586,455)	176,623	130,688
Benefit from (provision for) income taxes	4,608	(7,440)	(11,088)
Net (Loss) Income	$ (581,847)	$ 169,183	$ 119,600
Basic earnings per common share (in dollars per share)	$ (5.12)	$ 1.63	$ 1.47
Diluted earnings per common share (in dollars per share)	$ (5.12)	$ 1.46	$ 1.34
Basic weighted average shares outstanding (in shares)	113,935,605	101,120,744	78,724,912
Diluted weighted average shares outstanding (in shares)	113,935,605	136,780,594	110,027,770