Goodwill and Intangibles - Goodwill (Details) - USD ($)	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Goodwill
Decrease in goodwill	$ 1,600,000
Changes in goodwill:
Balance, beginning of period	17,477,000	$ 3,902,000	$ 523,000
Goodwill acquired during the period		13,163,000	3,379,000
Post-acquisition adjustments	(1,632,000)	412,000
Balance, end of the period	15,845,000	17,477,000	3,902,000
NattyMac, LLC
Goodwill
Transfer to intangible assets	1,600,000	1,600,000
Changes in goodwill:
Balance, beginning of period	3,700,000
Post-acquisition adjustments	271,000
Balance, end of the period		3,700,000
Multifamily
Changes in goodwill:
Balance, beginning of period	3,791,000	3,379,000
Goodwill acquired during the period			3,379,000
Post-acquisition adjustments		412,000
Balance, end of the period	3,791,000	3,791,000	3,379,000
Banking
Changes in goodwill:
Balance, beginning of period	8,686,000	523,000	523,000
Goodwill acquired during the period		8,163,000
Post-acquisition adjustments	(333,000)
Balance, end of the period	8,353,000	8,686,000	$ 523,000
Mortgage Warehousing
Changes in goodwill:
Balance, beginning of period	5,000,000
Goodwill acquired during the period		5,000,000
Post-acquisition adjustments	(1,299,000)
Balance, end of the period	$ 3,701,000	5,000,000
Mortgage Warehousing | NattyMac, LLC
Changes in goodwill:
Goodwill acquired during the period		$ 3,700,000