Other Investments - Additional Information (Details) - USD ($) $ in Thousands			1 Months Ended			2 Months Ended	3 Months Ended								9 Months Ended	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Mar. 31, 2019	Jan. 31, 2019	May 31, 2018	Feb. 28, 2019	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Sep. 30, 2019	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Investment [Line Items]
Commitment to fund partnership		$ 71,000
Equity method investment earnings																$ 11,000	$ 1,000
Investment fair value changes, net																35,500	(25,689)	$ 10,374
Interest income							$ 192,581	$ 150,117	$ 148,542	$ 131,041	$ 119,725	$ 99,397	$ 82,976	$ 76,619		622,281	378,717	248,057
Equity investment income																1,405	618	0
5 Arches, LLC
Investment [Line Items]
Minority interest, percentage					20.00%
Option to purchase additional equity, percent			80.00%
Light-Renovation Multifamily Loans
Investment [Line Items]
Commitment to fund partnership				$ 78,000
Commitment to fund partnership, funded amount	$ 41,000						41,000									41,000
Carrying amount of equity method investments	40,000						40,000									$ 40,000
Equity investment income															$ 1,000
5 Arches, LLC
Investment [Line Items]
Equity method investment earnings						$ 300											600
Payments to acquire equity method investment					$ 10,000
Option to purchase additional equity, term					1 year											1 year
Option to purchase additional equity, amount					$ 40,000
Amortization of intangible assets						$ 100											400
MSRs
Investment [Line Items]
Servicing asset, fair value	42,224	60,281					42,224				60,281					$ 42,224	60,281	$ 63,598	$ 118,526
Servicing assets	4,350,000	4,930,000					4,350,000				4,930,000					4,350,000	4,930,000
Shared home appreciation options
Investment [Line Items]
Investment fair value changes, net																842
Payments to acquire investments	43,000															45,000
Commitment To Fund Investment | Shared home appreciation options
Investment [Line Items]
Other commitment	7,000						7,000									7,000
Residential Loans
Investment [Line Items]
Servicing asset, fair value	169,204	300,468					169,204				300,468					169,204	300,468
Servicing asset, unpaid principal balance on underlying loan	7,940,000						7,940,000									7,940,000
Excess MSRs
Investment [Line Items]
Servicing asset, fair value	31,814	$ 27,312					31,814				$ 27,312					31,814	27,312
Investment fair value changes, net															(3,000)	2,000
Interest income															$ 8,000		1,000
Servicer advance financing
Investment [Line Items]
Collateral amounts	$ 152,000						$ 152,000									152,000
Investment fair value changes, net																$ 3,000	$ (1,000)