Other Assets - Schedule of Other Assets (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Other Assets [Abstract]
Contingent loan repurchase asset	$ 492,900	$ 302,581
Other prepaid expenses	21,996	27,647
Prepaid representation, warranty and indemnification claims - Agency MSR sale	15,173	15,173
Prepaid lender fees, net	8,647	6,589
REO	8,556	7,368
Derivatives, at fair value	6,007	4,552
Deferred tax assets, net	2,169	5,289
Security deposits	2,163	2,278
Mortgage-backed securities, at fair value	2,075	1,502
Interest-earning time deposits	390	1,338
Other	3,164	5,250
Other assets	$ 563,240	$ 379,567