Goodwill and Intangibles - Goodwill (Details) - USD ($)		12 Months Ended
	Dec. 31, 2018	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Changes in goodwill:
Balance, beginning of period		$ 15,845,000	$ 15,845,000	$ 15,845,000
Goodwill acquired during the period
Post-acquisition adjustments
Impairment losses
Balance, end of the period		15,845,000	15,845,000	15,845,000	$ 15,845,000
NattyMac, LLC
Goodwill
Transfer to intangible assets	$ 1,600,000				1,600,000
Changes in goodwill:
Post-acquisition adjustments					271,000
Multifamily
Changes in goodwill:
Balance, beginning of period		3,791,000	3,791,000	3,791,000
Goodwill acquired during the period
Post-acquisition adjustments
Impairment losses
Balance, end of the period		3,791,000	3,791,000	3,791,000	3,791,000
Banking
Changes in goodwill:
Balance, beginning of period		8,353,000	8,353,000	8,353,000
Goodwill acquired during the period
Post-acquisition adjustments
Impairment losses
Balance, end of the period		8,353,000	8,353,000	8,353,000	8,353,000
Mortgage Warehousing
Changes in goodwill:
Balance, beginning of period		3,701,000	3,701,000	3,701,000
Goodwill acquired during the period
Post-acquisition adjustments
Impairment losses
Balance, end of the period		$ 3,701,000	$ 3,701,000	$ 3,701,000	$ 3,701,000
Mortgage Warehousing | NattyMac, LLC
Changes in goodwill:
Goodwill acquired during the period	$ 3,700,000