Fair Value - Schedule of Carrying Amounts and Estimated Fair Values of Financial Instruments and Nonfinancial Assets Measured at Fair Value on a Recurring or Non-recurring basis or Disclosed, but not Carried, at Fair Value (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Financial assets:
Loans held for sale, at fair value	$ 208,752	$ 176,525	$ 214,262	$ 284,632
Financial liabilities:
Match funded liabilities	679,109	778,284
Financing liabilities:
HMBS-related borrowings	6,063,435	5,380,448
Other financing liabilities	972,595	1,062,090
Total Financing liabilities	1,030,306	452,736
Other secured borrowings:
Total Other secured borrowings	1,025,791	448,061
Senior notes:
Total Senior notes	311,085	448,727
Mortgage servicing rights:
Mortgage servicing rights, at fair value	1,486,395	1,457,149	671,962	$ 679,256
Financing Liability - MSRs Pledged
Financing liabilities:
Other financing liabilities	950,593	1,032,856	$ 508,291
Carrying Value
Financial assets:
Total Loans held for sale	275,269	242,622
Loans held for investment	6,292,938	5,498,719
Financing liabilities:
Total Financing liabilities	7,036,030	6,442,538
Other secured borrowings:
Total Other secured borrowings	1,025,791	448,061
Senior notes:
Total Senior notes	311,085	448,727
Fair Value
Financial assets:
Total Loans held for sale	275,269	242,622
Loans held for investment	6,292,938	5,498,719
Financing liabilities:
Total Financing liabilities	7,036,030	6,442,538
Other secured borrowings:
Total Other secured borrowings	1,010,789	432,313
Senior notes:
Total Senior notes	270,022	426,147
Level 2 | Carrying Value
Financial assets:
Loans held for sale, at fair value	208,752	176,525
Corporate bonds	441	450
Other secured borrowings:
Senior secured term loan	322,758	226,825
Senior notes:
Senior unsecured notes	21,046	119,924
Senior secured notes	290,039	328,803
Derivative financial instrument assets (liabilities), at fair value
Interest rate lock commitments	4,878	3,871
TBA / Forward mortgage-backed securities (MBS) trades - MSR hedging	1,121	0
Level 2 | Fair Value
Financial assets:
Loans held for sale, at fair value	208,752	176,525
Corporate bonds	441	450
Other secured borrowings:
Senior secured term loan	324,643	227,449
Senior notes:
Senior unsecured notes	13,821	119,258
Senior secured notes	256,201	306,889
Derivative financial instrument assets (liabilities), at fair value
Interest rate lock commitments	4,878	3,871
TBA / Forward mortgage-backed securities (MBS) trades - MSR hedging	1,121	0
Level 3 | Carrying Value
Financial assets:
Loans held for sale, at lower of cost or fair value	66,517	66,097
Loans held for investment	6,269,596	5,472,199
Advances (including match funded)	1,056,523	1,186,676
Receivables, net	201,220	198,262
Mortgage-backed securities, at fair value	2,075	1,502
Financial liabilities:
Match funded liabilities	679,109	778,284
Financing liabilities:
HMBS-related borrowings	6,063,435	5,380,448
Other secured borrowings:
Other	703,033	221,236
Derivative financial instrument assets (liabilities), at fair value
Interest rate caps	0	678
Mortgage servicing rights:
Mortgage servicing rights, at fair value	1,486,395	1,457,149
Level 3 | Carrying Value | Loans Held for Investments - Restricted for Securitization Investors
Financial assets:
Loans held for investment	23,342	26,520
Level 3 | Carrying Value | Financing Liability - MSRs Pledged
Financing liabilities:
Other financing liabilities	950,593	1,032,856
Level 3 | Carrying Value | Financing Liability Owed to Securitization Investors
Financing liabilities:
Other financing liabilities	22,002	24,815
Level 3 | Carrying Value | Other Financing Liabilities With Unrelated Parties
Financing liabilities:
Other financing liabilities	0	4,419
Level 3 | Fair Value
Financial assets:
Loans held for sale, at lower of cost or fair value	66,517	66,097
Loans held for investment	6,269,596	5,472,199
Advances (including match funded)	1,056,523	1,186,676
Receivables, net	201,220	198,262
Mortgage-backed securities, at fair value	2,075	1,502
Financial liabilities:
Match funded liabilities	679,507	776,485
Financing liabilities:
HMBS-related borrowings	6,063,435	5,380,448
Other secured borrowings:
Other	686,146	204,864
Derivative financial instrument assets (liabilities), at fair value
Interest rate caps	0	678
Mortgage servicing rights:
Mortgage servicing rights, at fair value	1,486,395	1,457,149
Level 3 | Fair Value | Loans Held for Investments - Restricted for Securitization Investors
Financial assets:
Loans held for investment	23,342	26,520
Level 3 | Fair Value | Financing Liability - MSRs Pledged
Financing liabilities:
Other financing liabilities	950,593	1,032,856
Level 3 | Fair Value | Financing Liability Owed to Securitization Investors
Financing liabilities:
Other financing liabilities	22,002	24,815
Level 3 | Fair Value | Other Financing Liabilities With Unrelated Parties
Financing liabilities:
Other financing liabilities	0	4,419
Level 1 | Carrying Value
Financial assets:
U.S. Treasury notes	0	1,064
Derivative financial instrument assets (liabilities), at fair value
Forward mortgage-backed securities trades	(92)	(4,983)
Level 1 | Fair Value
Financial assets:
U.S. Treasury notes	0	1,064
Derivative financial instrument assets (liabilities), at fair value
Forward mortgage-backed securities trades	$ (92)	$ (4,983)