Fair Value Measurements - Financial Instruments Measured at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Jun. 30, 2022	Dec. 31, 2021
Financial assets:
Loans, at fair value	$ 29,484	$ 29,697
Derivative Asset, Subject to Master Netting Arrangement, before Offset	1,216	250
Financial liabilities:
Derivative Liability	1,887	228
Fair Value, Measurements, Recurring
Financial assets:
MSR financing receivables	120,260	125,018
Loans, at fair value	29,484	29,697
Derivative Asset, Subject to Master Netting Arrangement, before Offset	1,216	250
Other assets	5,746	12,655
Financial liabilities:
Derivative Liability	1,887	228
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 1
Financial assets:
MSR financing receivables	0	0
Loans, at fair value	0	0
Derivative Asset, Subject to Master Netting Arrangement, before Offset	0	20
Other assets	511	5,267
Financial liabilities:
Derivative Liability	0	0
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 2
Financial assets:
MSR financing receivables	0	0
Loans, at fair value	0	0
Derivative Asset, Subject to Master Netting Arrangement, before Offset	1,216	230
Other assets	0	0
Financial liabilities:
Derivative Liability	1,887	228
Fair Value, Measurements, Recurring | Fair Value, Inputs, Level 3
Financial assets:
MSR financing receivables	120,260	125,018
Loans, at fair value	29,484	29,697
Derivative Asset, Subject to Master Netting Arrangement, before Offset	0	0
Other assets	5,235	7,388
Financial liabilities:
Derivative Liability	0	0
Fair Value, Measurements, Recurring | Agency MBS
Financial assets:
Trading securities	382,357	483,927
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 1
Financial assets:
Trading securities	0	0
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 2
Financial assets:
Trading securities	382,357	483,927
Fair Value, Measurements, Recurring | Agency MBS | Fair Value, Inputs, Level 3
Financial assets:
Trading securities	0	0
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary
Financial assets:
Loans, at fair value	225,004	7,442
Financial liabilities:
Secured debt of consolidated VIEs, at fair value	205,497	508
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 1
Financial assets:
Loans, at fair value	0	0
Financial liabilities:
Secured debt of consolidated VIEs, at fair value	0	0
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 2
Financial assets:
Loans, at fair value	0	0
Financial liabilities:
Secured debt of consolidated VIEs, at fair value	193,976	0
Fair Value, Measurements, Recurring | Variable Interest Entity, Primary Beneficiary | Fair Value, Inputs, Level 3
Financial assets:
Loans, at fair value	225,004	7,442
Financial liabilities:
Secured debt of consolidated VIEs, at fair value	11,521	508
Fair Value, Measurements, Recurring | Credit Securities
Financial assets:
Trading securities	113,419	26,222
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 1
Financial assets:
Trading securities	0	0
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 2
Financial assets:
Trading securities	99,901	0
Fair Value, Measurements, Recurring | Credit Securities | Fair Value, Inputs, Level 3
Financial assets:
Trading securities	$ 13,518	$ 26,222