Segment Reporting (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($) segment	Dec. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)
Segment Reporting Information [Line Items]
Number of reportable segments | segment	3
Revenues:
Interest income	$ 95,480	$ 50,015	$ 43,666
Gain on sale of loans	20,080	25,164	19,633
Other revenues	12,012	6,639	6,432
Total operating income	127,572	81,818	69,731
Expenses:
Interest expense	11,108	6,304	4,119
Salaries and employee benefits	43,915	31,778	26,966
Other expenses	44,072	18,132	15,050
Total operating expenses	99,095	56,214	46,135
Income (loss) before income taxes	28,477	25,604	23,596
Total assets	2,873,894	1,430,708	1,178,548
Capital expenditures	1,594	871	300
Operating Segments [Member] | Commercial Mortgage Backed Securities [Member]
Revenues:
Interest income	94,577	49,063	42,763
Gain on sale of loans	136	0	0
Other revenues	6,270	3,893	3,229
Total operating income	100,983	52,956	45,992
Expenses:
Interest expense	10,912	6,324	4,135
Salaries and employee benefits	28,108	16,015	13,519
Other expenses	28,998	9,232	7,732
Total operating expenses	68,018	31,571	25,386
Income (loss) before income taxes	32,965	21,385	20,606
Total assets	2,827,041	1,394,061	1,133,916
Capital expenditures	1,566	826	252
Operating Segments [Member] | Mortgage Banking [Member]
Revenues:
Interest income	1,141	1,767	1,650
Gain on sale of loans	19,944	25,164	19,633
Other revenues	(307)	(424)	388
Total operating income	20,778	26,507	21,671
Expenses:
Interest expense	(6)	548	467
Salaries and employee benefits	11,958	13,541	11,470
Other expenses	4,338	5,354	5,087
Total operating expenses	16,290	19,443	17,024
Income (loss) before income taxes	4,488	7,064	4,647
Total assets	31,999	39,356	46,077
Capital expenditures	0	3	15
Operating Segments [Member] | Wealth Management [Member]
Revenues:
Interest income	7	0	0
Gain on sale of loans	0	0	0
Other revenues	5,988	3,034	2,671
Total operating income	5,995	3,034	2,671
Expenses:
Interest expense	0	0	0
Salaries and employee benefits	3,849	2,222	1,977
Other expenses	3,460	1,034	1,116
Total operating expenses	7,309	3,256	3,093
Income (loss) before income taxes	(1,314)	(222)	(422)
Total assets	10,967	2,841	3,205
Capital expenditures	1	2	24
Operating Segments [Member] | Other [Member]
Revenues:
Interest income	25	20	16
Gain on sale of loans	0	0	0
Other revenues	1,453	1,401	1,391
Total operating income	1,478	1,421	1,407
Expenses:
Interest expense	472	267	280
Salaries and employee benefits	0	0	0
Other expenses	8,668	3,777	2,362
Total operating expenses	9,140	4,044	2,642
Income (loss) before income taxes	(7,662)	(2,623)	(1,235)
Total assets	21,727	18,037	16,837
Capital expenditures	27	40	9
Intersegment Eliminations [Member]
Revenues:
Interest income	(270)	(835)	(763)
Gain on sale of loans	0	0	0
Other revenues	(1,392)	(1,265)	(1,247)
Total operating income	(1,662)	(2,100)	(2,010)
Expenses:
Interest expense	(270)	(835)	(763)
Salaries and employee benefits	0	0	0
Other expenses	(1,392)	(1,265)	(1,247)
Total operating expenses	(1,662)	(2,100)	(2,010)
Income (loss) before income taxes	0	0	0
Total assets	(17,840)	(23,587)	(21,487)
Capital expenditures	$ 0	$ 0	$ 0