Discontinued Operations and Assets and Liabilities Held For Sale (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021
Assets:
Cash and cash equivalents	$ 13,694	$ 15,642
Restricted cash	6,314	7,781
Loans, net	2,778	4,511
Loans, held for sale	133,204	134,642
Loans eligible for repurchase from Ginnie Mae	86,872	66,193
Derivative instruments	847	431
Servicing rights	188,855	192,203
Other assets	22,032	17,788
Assets	454,596	439,191
Liabilities:
Secured borrowings	230,965	182,558
Liabilities for loans eligible for repurchase from Ginnie Mae	86,872	66,193
Derivative instruments	1,321	267
Accounts payable and other accrued liabilities	13,999	22,906
Total Liabilities	333,157	271,924
Interest income	7,148	7,953	$ 8,300
Interest expense	(7,655)	(8,414)	(9,193)
Net interest income (loss) before provision for loan losses	(507)	(461)	(893)
Non-interest income
Residential mortgage banking activities	33,439	23,973	137,297
Net unrealized gain (loss) on financial instruments	(15,426)	46,063	16,921
Servicing income, net of amortization and impairment	37,181	34,497	30,392
Other income	47	38	2,153
Total non-interest income	55,241	104,571	186,763
Non-interest expense
Employee compensation and benefits	(19,177)	(24,237)	(32,973)
Variable expenses on residential mortgage banking activities	(21,822)	(4,340)	(75,133)
Professional fees	(641)	(791)	(2,951)
Loan servicing expense	(10,130)	(9,222)	(9,417)
Other operating expenses	(6,743)	(7,650)	(8,499)
Total non-interest expense	(58,513)	(46,240)	(128,973)
Income (loss) from discontinued operations before provision for income taxes	(3,779)	57,870	56,897
Income tax benefit (provision)	945	(14,258)	(14,224)
Net income (loss) from discontinued operations	$ (2,834)	$ 43,612	$ 42,673