Segment Information - Balance Sheet (Details) - USD ($) $ in Thousands	Sep. 30, 2022	Dec. 31, 2021	Sep. 30, 2021	Dec. 31, 2020
Assets:
Cash and cash equivalents	$ 389,651	$ 404,580	$ 380,730	$ 339,528
Restricted cash	922,531	486,690	$ 569,928	$ 197,470
Loans and investments, net	14,791,426	11,981,048
Loans held-for-sale, net	543,876	1,093,609
Capitalized mortgage servicing rights, net	403,886	422,734
Securities held-to-maturity, net	157,818	140,484
Investments in equity affiliates	84,047	89,676
Goodwill and other intangible assets	97,242	100,760
Other assets and due from related party	371,652	354,264
Total assets	17,762,129	15,073,845
Liabilities:
Debt obligations	14,377,628	12,056,701
Allowance for loss-sharing obligations	53,511	56,064
Other liabilities and due to related parties	376,984	411,096
Total liabilities	14,808,123	12,523,861
Structured Business | Operating segments
Assets:
Cash and cash equivalents	119,793	142,771
Restricted cash	903,587	468,013
Loans and investments, net	14,791,426	11,981,048
Investments in equity affiliates	84,047	89,676
Goodwill and other intangible assets	12,500	12,500
Other assets and due from related party	293,252	285,600
Total assets	16,204,605	12,979,608
Liabilities:
Debt obligations	13,866,114	11,100,429
Other liabilities and due to related parties	253,390	278,726
Total liabilities	14,119,504	11,379,155
Agency Business | Operating segments
Assets:
Cash and cash equivalents	269,858	261,809
Restricted cash	18,944	18,677
Loans held-for-sale, net	543,876	1,093,609
Capitalized mortgage servicing rights, net	403,886	422,734
Securities held-to-maturity, net	157,818	140,484
Goodwill and other intangible assets	84,742	88,260
Other assets and due from related party	78,400	68,664
Total assets	1,557,524	2,094,237
Liabilities:
Debt obligations	511,514	956,272
Allowance for loss-sharing obligations	53,511	56,064
Other liabilities and due to related parties	123,594	132,370
Total liabilities	$ 688,619	$ 1,144,706