Fair Value - Schedule of Significant Assumptions used in Valuation (Details)	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)
Measurement Input, Expected Term | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	6.00	5.90
Measurement Input, Expected Term | Automotive Dealer Financing Notes
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs		0.22
Measurement Input, Expected Term | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	6.00	5.90
Measurement Input, Prepayment Rate | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.146	0.147
Measurement Input, Prepayment Rate | Fair Value Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.117	0.085
Measurement Input, Prepayment Rate | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.122	0.154
Measurement Input, Prepayment Rate | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.146	0.147
Measurement Input, Prepayment Rate | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.119	0.139
Measurement Input, Default Rate | Fair Value Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.032	0.066
Measurement Input, Default Rate | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.273	0.271
Measurement Input, Default Rate | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.203	0.203
Measurement Input, Discount Rate | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.028	0.034
Measurement Input, Discount Rate | Fair Value Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.093	0.091
Measurement Input, Discount Rate | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.113	0.128
Measurement Input, Discount Rate | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.027	0.0334
Measurement Input, Discount Rate | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.107	0.120
Measurement Input, Weighted Average Cost To Service, In USD Per Service | Fair Value Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	85	90
Measurement Input, Weighted Average Cost To Service, In USD Per Service | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	277	297
Measurement Input, Weighted Average Cost To Service, In USD Per Service | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	223	234
5-Year Swap Rate | Measurement Input, Advance Financing Cost, Basis Spread On Variable Rate | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.02	0.0275
5-Year Swap Rate | Measurement Input, Advance Financing Cost, Basis Spread On Variable Rate | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.0200	0.0275
5-Year Swap Rate | Measurement Input, Interest Rate For Computing Float Earnings, Basis Spread On Variable Rate | Fair Value Non-Agency Mortgage Servicing Rights
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.0050	0.0050
5-Year Swap Rate | Measurement Input, Interest Rate For Computing Float Earnings, Basis Spread On Variable Rate | Mortgage Servicing Rights Pledged
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.0050	0.005
Minimum | Measurement Input, Expected Term | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	2.40	3.00
Minimum | Measurement Input, Expected Term | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	2.40	3.00
Minimum | Measurement Input, Prepayment Rate | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.0780	0.068
Minimum | Measurement Input, Prepayment Rate | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.0780	0.068
Maximum | Measurement Input, Expected Term | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	7.80	7.60
Maximum | Measurement Input, Expected Term | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	7.80	7.60
Maximum | Measurement Input, Prepayment Rate | Loans Held for Investment
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.2830	0.384
Maximum | Measurement Input, Prepayment Rate | HMBS - Related Borrowings
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Measurement inputs	0.2830	0.384