Loans - Impaired Loans with Associated Allowance (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Unpaid principal balance
Unpaid principal balance, with no related allowance	$ 2,341	$ 1,815
Unpaid principal balance, with related allowance	1,454	1,346
Unpaid principal balance	3,795	3,161
Recorded investment
Recorded investment, with no related allowance	1,999	1,234
Recorded investment, with related allowance	916	813
Recorded investment	2,915	2,047
Specific reserves	296	280
Loans receivable, nonaccrual status	7,758	7,237
SBA loans held for investment
Unpaid principal balance
Unpaid principal balance, with no related allowance	948	1,235
Unpaid principal balance, with related allowance	1,098	975
Unpaid principal balance	2,046	2,210
Recorded investment
Recorded investment, with no related allowance	606	653
Recorded investment, with related allowance	573	455
Recorded investment	1,179	1,108
Specific reserves	277	246
Loans receivable, nonaccrual status	1,239	1,168
SBA 504 loans
Unpaid principal balance
Unpaid principal balance, with no related allowance	237	513
Unpaid principal balance	237	513
Recorded investment
Recorded investment, with no related allowance	237	513
Recorded investment	237	513
Specific reserves	0
Loans receivable, nonaccrual status	237	513
Commercial loans
Unpaid principal balance
Unpaid principal balance, with no related allowance	1,156	67
Unpaid principal balance, with related allowance	356	371
Unpaid principal balance	1,512	438
Recorded investment
Recorded investment, with no related allowance	1,156	68
Recorded investment, with related allowance	343	358
Recorded investment	1,499	426
Specific reserves	19	34
Commercial loans | Other
Unpaid principal balance
Unpaid principal balance, with no related allowance	25	25
Unpaid principal balance, with related allowance	13	13
Unpaid principal balance	38	38
Recorded investment
Recorded investment, with no related allowance	25	25
Recorded investment, with related allowance	0	0
Recorded investment	25	25
Specific reserves	0	0
Loans receivable, nonaccrual status	25	25
Commercial loans | Commercial real estate
Unpaid principal balance
Unpaid principal balance, with no related allowance	1,131	42
Unpaid principal balance, with related allowance	343	358
Unpaid principal balance	1,474	400
Recorded investment
Recorded investment, with no related allowance	1,131	43
Recorded investment, with related allowance	343	358
Recorded investment	1,474	401
Specific reserves	19	34
Loans receivable, nonaccrual status	1,474	401
Commercial loans | Commercial real estate construction
Recorded investment
Loans receivable, nonaccrual status	0	0
Small Business Administration
Recorded investment
Loans receivable, nonaccrual status	$ 60	$ 60