Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Segment Reporting Information [Line Items]
Interest income	$ 192,581	$ 150,117	$ 148,542	$ 131,041	$ 119,725	$ 99,397	$ 82,976	$ 76,619	$ 622,281	$ 378,717	$ 248,057
Interest expense	(147,708)	(116,604)	(116,220)	(99,276)	(84,961)	(64,351)	(48,213)	(41,514)	(479,808)	(239,039)	(108,816)
Net Interest Income	44,873	33,513	32,322	31,765	34,764	35,046	34,763	35,105	142,473	139,678	139,241
Non-interest income
Mortgage banking activities, net									87,266	59,566	53,908
Investment fair value changes, net									35,500	(25,689)	10,374
Other income									19,257	13,070	12,436
Realized gains, net									23,821	27,041	13,355
Total non-interest income, net	58,052	30,029	29,984	47,779	(17,538)	32,327	19,527	39,672	165,844	73,988	90,073
General and administrative expenses									(118,672)	(82,782)	(77,156)
Other expenses									(13,022)	(196)	0
Provision for income taxes									(7,440)	(11,088)	(11,752)
Net Income	$ 49,143	$ 34,310	$ 31,266	$ 54,464	$ (913)	$ 40,921	$ 32,747	$ 46,845	169,183	119,600	140,406
Non-cash amortization income (expense), net									(3,361)	12,379	17,462
Operating Segments | Residential Lending
Segment Reporting Information [Line Items]
Interest income									268,559	245,124	158,819
Interest expense									(192,359)	(158,498)	(66,364)
Net Interest Income									76,200	86,626	92,455
Non-interest income
Mortgage banking activities, net									47,743	59,623	53,908
Investment fair value changes, net									(27,920)	(21,686)	(25,466)
Other income									9,210	12,452	12,436
Realized gains, net									8,292	7,709	3,907
Total non-interest income, net									37,325	58,098	44,785
General and administrative expenses									(30,671)	(32,139)	(28,622)
Other expenses									0	0
Provision for income taxes									(4,074)	(8,033)	(6,641)
Net Income									78,780	104,552	101,977
Non-cash amortization income (expense), net									3,669	4,486	4,946
Operating Segments | Business Purpose Lending
Segment Reporting Information [Line Items]
Interest income									54,372	4,588	0
Interest expense									(35,663)	(2,252)	0
Net Interest Income									18,709	2,336	0
Non-interest income
Mortgage banking activities, net									39,523	(57)	0
Investment fair value changes, net									(6,722)	(29)	0
Other income									5,852	0	0
Realized gains, net									0	0	0
Total non-interest income, net									38,653	(86)	0
General and administrative expenses									(30,655)	(2,597)	0
Other expenses									(8,521)	0
Provision for income taxes									(947)	0	0
Net Income									17,239	(347)	0
Non-cash amortization income (expense), net									(9,173)	(290)	0
Operating Segments | Multifamily Investments
Segment Reporting Information [Line Items]
Interest income									161,692	40,050	9,170
Interest expense									(151,980)	(34,324)	(6,107)
Net Interest Income									9,712	5,726	3,063
Non-interest income
Mortgage banking activities, net									0	0	0
Investment fair value changes, net									27,097	3,979	16,196
Other income									1,484	0	0
Realized gains, net									134	0	0
Total non-interest income, net									28,715	3,979	16,196
General and administrative expenses									(1,498)	(869)	(425)
Other expenses									0	0
Provision for income taxes									(11)	(16)	(238)
Net Income									36,918	8,820	18,596
Non-cash amortization income (expense), net									(1)	(1)	(1)
Operating Segments | Third-Party Residential Investments
Segment Reporting Information [Line Items]
Interest income									120,009	68,919	60,661
Interest expense									(85,388)	(27,446)	(21,512)
Net Interest Income									34,621	41,473	39,149
Non-interest income
Mortgage banking activities, net									0	0	0
Investment fair value changes, net									44,662	(6,957)	27,684
Other income									0	0	0
Realized gains, net									15,395	19,332	10,200
Total non-interest income, net									60,057	12,375	37,884
General and administrative expenses									(2,843)	(2,855)	(2,028)
Other expenses									(1,106)	(18)
Provision for income taxes									(2,408)	(3,039)	(4,873)
Net Income									88,321	47,936	70,132
Non-cash amortization income (expense), net									6,957	12,295	15,927
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income									17,649	20,036	19,407
Interest expense									(14,418)	(16,519)	(14,833)
Net Interest Income									3,231	3,517	4,574
Non-interest income
Mortgage banking activities, net									0	0	0
Investment fair value changes, net									(1,617)	(996)	(8,040)
Other income									2,711	618	0
Realized gains, net									0	0	(752)
Total non-interest income, net									1,094	(378)	(8,792)
General and administrative expenses									(53,005)	(44,322)	(46,081)
Other expenses									(3,395)	(178)	0
Provision for income taxes									0	0	0
Net Income									(52,075)	(41,361)	(50,299)
Non-cash amortization income (expense), net									$ (4,813)	$ (4,111)	$ (3,410)