Short-Term Debt - Collateral for Short-Term Debt Facilities (Details) - USD ($) $ in Thousands	Dec. 31, 2022	Dec. 31, 2021
Short-term Debt [Line Items]
Total Collateral for Short-Term Debt Facilities	$ 2,299,034	$ 2,450,304
Line of Credit
Short-term Debt [Line Items]
Total real estate securities owned	178,070	99,005
Restricted cash and other assets	1,097	1,962
Total Collateral for Short-Term Debt Facilities	2,017,062	2,107,451
Servicer advance financing
Short-term Debt [Line Items]
Total Collateral for Short-Term Debt Facilities	281,972	342,853
Cash	12,713	6,480
Restricted cash	0	25,420
Servicer advances	269,259	310,953
Residential loans, held-for-sale, at fair value | Line of Credit
Short-term Debt [Line Items]
Short term debt, restricted	775,545	1,838,797
Business purpose loans | Line of Credit
Short-term Debt [Line Items]
Short term debt, restricted	871,072	167,687
HEI | Line of Credit
Short-term Debt [Line Items]
Short term debt, restricted	191,278	0
On balance sheet | Line of Credit
Short-term Debt [Line Items]
On balance sheet	72,133	5,823
Sequoia | Sequoia securities | Short Term Borrowing Agreement | Line of Credit
Short-term Debt [Line Items]
Trading securities pledged as collateral	74,170	61,525
Freddie Mac K-Series | Sequoia securities | Short Term Borrowing Agreement | Line of Credit
Short-term Debt [Line Items]
Trading securities pledged as collateral	$ 31,767	$ 31,657