Segment Information (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Segment Information
Interest income	$ 63,799	$ 59,761	$ 48,761	$ 39,674	$ 39,825	$ 37,577	$ 34,123	$ 29,038	$ 211,995	$ 140,563	$ 94,387
Interest expense	26,178	27,137	20,839	15,543	15,650	14,095	11,917	8,930	89,697	50,592	27,790
Net interest income	37,621	32,624	27,922	24,131	24,175	23,482	22,206	20,108	122,298	89,971	66,597
Provision for loan losses	1,993	1,193	105	649	1,608	617	998	1,406	3,940	4,629	2,472
Net Interest Income After Provision for Loan Losses	35,628	31,431	27,817	23,482	22,567	22,865	21,208	18,702	118,358	85,342	64,125
Noninterest income	22,703	10,852	9,870	3,664	14,735	11,907	11,630	11,313	47,089	49,585	47,680
Noninterest expense	18,836	15,522	15,920	13,035	16,181	12,449	12,000	10,270	63,313	50,900	34,644
Income Before Income Taxes	39,495	26,761	21,767	14,111	21,121	22,323	20,838	19,745	102,134	84,027	77,161
Income taxes	9,434	6,502	5,328	3,541	5,699	5,584	5,186	4,684	24,805	21,153	22,477
Net Income	30,061	$ 20,259	$ 16,439	$ 10,570	15,422	$ 16,739	$ 15,652	$ 15,061	77,329	62,874	54,684
Total assets	6,371,928				3,884,163				6,371,928	3,884,163	3,393,133
Other
Segment Information
Interest income									2,067	1,735	721
Interest expense									(6,864)	(3,285)	(736)
Net interest income									8,931	5,020	1,457
Net Interest Income After Provision for Loan Losses									8,931	5,020	1,457
Noninterest income									(2,776)	(1,946)	(814)
Noninterest expense									11,622	9,098	6,188
Income Before Income Taxes									(5,467)	(6,024)	(5,545)
Income taxes									(1,413)	(1,819)	(1,917)
Net Income									(4,054)	(4,205)	(3,628)
Total assets	39,810				30,598				39,810	30,598	16,855
Multifamily | Operating Segments
Segment Information
Interest income									1,328	712	414
Net interest income									1,328	712	414
Net Interest Income After Provision for Loan Losses									1,328	712	414
Noninterest income									41,682	45,831	43,715
Noninterest expense									22,556	19,205	10,911
Income Before Income Taxes									20,454	27,338	33,218
Income taxes									5,691	7,528	4,557
Net Income									14,763	19,810	28,661
Total assets	188,866				166,102				188,866	166,102	134,390
Mortgage Warehousing | Operating Segments
Segment Information
Interest income									102,157	58,784	48,798
Interest expense									50,880	24,369	13,673
Net interest income									51,277	34,415	35,125
Provision for loan losses									1,358	1,372	452
Net Interest Income After Provision for Loan Losses									49,919	33,043	34,673
Noninterest income									7,178	2,550	2,836
Noninterest expense									11,397	7,721	7,710
Income Before Income Taxes									45,700	27,872	29,799
Income taxes									10,934	6,872	11,558
Net Income									34,766	21,000	18,241
Total assets	3,124,684				1,430,776				3,124,684	1,430,776	1,352,748
Banking | Operating Segments
Segment Information
Interest income									106,443	79,332	44,454
Interest expense									45,681	29,508	14,853
Net interest income									60,762	49,824	29,601
Provision for loan losses									2,582	3,257	2,020
Net Interest Income After Provision for Loan Losses									58,180	46,567	27,581
Noninterest income									1,005	3,150	1,943
Noninterest expense									17,738	14,876	9,835
Income Before Income Taxes									41,447	34,841	19,689
Income taxes									9,593	8,572	8,279
Net Income									31,854	26,269	11,410
Total assets	$ 3,018,568				$ 2,256,687				$ 3,018,568	$ 2,256,687	$ 1,889,140