Securitizations and Variable Interests Entities - Schedule of Carrying Value of Assets and Liabilities of Consolidated Mortgage-backed Securitization Trusts (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Qualitative and Quantitative Information, Transferor's Continuing Involvement [Line Items]
Loans held for investment	$ 6,292,938	$ 5,498,719
Other financing liabilities	972,595	1,062,090
Residential Mortgage Backed Securitization Trusts
Qualitative and Quantitative Information, Transferor's Continuing Involvement [Line Items]
Loans held for investment	23,342	26,520
Other financing liabilities	$ 22,002	$ 24,815