Loans Receivable and Allowance for Loan Losses - Past Due Financing Receivables (Details) (USD $)	Jun. 30, 2014		Dec. 31, 2013
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	$ 9,573,000		$ 7,322,000
60 - 89 Days Past Due	850,000		2,772,000
Greater than 90 Days Past Due	0		5,000
Total Past Due	10,423,000		10,099,000
Non-accrual loans	42,651,000	[1]	39,154,000	[2]
Total Delinquent loans	53,074,000		49,253,000
Guaranteed portion of delinquent SBA loans excluded from Nonaccrual loans	30,000,000		27,500,000
Legacy
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	3,170,000		2,209,000
60 - 89 Days Past Due	210,000		266,000
Greater than 90 Days Past Due	0		0
Total Past Due	3,380,000		2,475,000
Non-accrual loans	22,973,000	[1]	18,440,000	[2]
Total Delinquent loans	26,353,000		20,915,000
Legacy | Real estate | Residential
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	0		0
Non-accrual loans	0	[1]	0	[2]
Total Delinquent loans	0		0
Legacy | Real estate | Commercial | Retail
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	1,170,000		122,000
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	1,170,000		122,000
Non-accrual loans	3,917,000	[1]	4,363,000	[2]
Total Delinquent loans	5,087,000		4,485,000
Legacy | Real estate | Commercial | Hotel & motel
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	284,000		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	284,000		0
Non-accrual loans	121,000	[1]	121,000	[2]
Total Delinquent loans	405,000		121,000
Legacy | Real estate | Commercial | Gas station & car wash
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0		1,038,000
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	0		1,038,000
Non-accrual loans	4,026,000	[1]	2,228,000	[2]
Total Delinquent loans	4,026,000		3,266,000
Legacy | Real estate | Commercial | Mixed use
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	0		0
Non-accrual loans	966,000	[1]	974,000	[2]
Total Delinquent loans	966,000		974,000
Legacy | Real estate | Commercial | Industrial & warehouse
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	211,000		215,000
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	211,000		215,000
Non-accrual loans	2,020,000	[1]	1,923,000	[2]
Total Delinquent loans	2,231,000		2,138,000
Legacy | Real estate | Commercial | Other
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	10,000		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	10,000		0
Non-accrual loans	529,000	[1]	1,398,000	[2]
Total Delinquent loans	539,000		1,398,000
Legacy | Real estate | Construction
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	0		0
Non-accrual loans	0	[1]	0	[2]
Total Delinquent loans	0		0
Legacy | Commercial business
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	1,428,000		780,000
60 - 89 Days Past Due	182,000		244,000
Greater than 90 Days Past Due	0		0
Total Past Due	1,610,000		1,024,000
Non-accrual loans	7,409,000	[1]	6,402,000	[2]
Total Delinquent loans	9,019,000		7,426,000
Legacy | Trade finance
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	30,000		0
60 - 89 Days Past Due	0		0
Greater than 90 Days Past Due	0		0
Total Past Due	30,000		0
Non-accrual loans	3,985,000	[1]	1,031,000	[2]
Total Delinquent loans	4,015,000		1,031,000
Legacy | Consumer and other
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	37,000		54,000
60 - 89 Days Past Due	28,000		22,000
Greater than 90 Days Past Due	0		0
Total Past Due	65,000		76,000
Non-accrual loans	0	[1]	0	[2]
Total Delinquent loans	65,000		76,000
Acquired
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	6,403,000	[3]	5,113,000	[3]
60 - 89 Days Past Due	640,000	[3]	2,506,000	[3]
Greater than 90 Days Past Due	0	[3]	5,000	[3]
Total Past Due	7,043,000	[3]	7,624,000	[3]
Non-accrual loans	19,678,000	[1],[3]	20,714,000	[2],[3]
Total Delinquent loans	26,721,000	[3]	28,338,000	[3]
Acquired | Real estate | Residential
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	0	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due		[3]	0	[3]
Total Past Due	0	[3]	0	[3]
Non-accrual loans	0	[1],[3]	0	[3]
Total Delinquent loans	0	[3]	0	[3]
Acquired | Real estate | Commercial | Retail
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	62,000	[3]	2,024,000	[3]
60 - 89 Days Past Due	73,000	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	135,000	[3]	2,024,000	[3]
Non-accrual loans	1,268,000	[1],[3]	1,030,000	[3]
Total Delinquent loans	1,403,000	[3]	3,054,000	[3]
Acquired | Real estate | Commercial | Hotel & motel
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	0	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	0	[3]	0	[3]
Non-accrual loans	6,317,000	[1],[3]	6,441,000	[3]
Total Delinquent loans	6,317,000	[3]	6,441,000	[3]
Acquired | Real estate | Commercial | Gas station & car wash
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	1,068,000	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	0	[3]	1,068,000	[3]
Non-accrual loans	1,691,000	[1],[3]	1,339,000	[3]
Total Delinquent loans	1,691,000	[3]	2,407,000	[3]
Acquired | Real estate | Commercial | Mixed use
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	5,652,000	[3]	576,000	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	5,652,000	[3]	576,000	[3]
Non-accrual loans	809,000	[1],[3]	0	[3]
Total Delinquent loans	6,461,000	[3]	576,000	[3]
Acquired | Real estate | Commercial | Industrial & warehouse
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	121,000	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	0	[3]	121,000	[3]
Non-accrual loans	1,301,000	[1],[3]	6,890,000	[3]
Total Delinquent loans	1,301,000	[3]	7,011,000	[3]
Acquired | Real estate | Commercial | Other
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	516,000	[3]
60 - 89 Days Past Due	264,000	[3]	1,729,000	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	264,000	[3]	2,245,000	[3]
Non-accrual loans	4,850,000	[1],[3]	1,376,000	[3]
Total Delinquent loans	5,114,000	[3]	3,621,000	[3]
Acquired | Real estate | Construction
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	0	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	0	[3]	0	[3]
Non-accrual loans	0	[1],[3]	0	[3]
Total Delinquent loans	0	[3]	0	[3]
Acquired | Commercial business
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	684,000	[3]	524,000	[3]
60 - 89 Days Past Due	176,000	[3]	703,000	[3]
Greater than 90 Days Past Due	0	[3]	5,000	[3]
Total Past Due	860,000	[3]	1,232,000	[3]
Non-accrual loans	2,159,000	[1],[3]	2,708,000	[3]
Total Delinquent loans	3,019,000	[3]	3,940,000	[3]
Acquired | Trade finance
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	0	[3]	0	[3]
60 - 89 Days Past Due	0	[3]	0	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	0	[3]	0	[3]
Non-accrual loans	0	[1],[3]	0	[3]
Total Delinquent loans	0	[3]	0	[3]
Acquired | Consumer and other
Financing Receivable, Recorded Investment, Past Due [Line Items]
30 - 59 Days Past Due	5,000	[3]	284,000	[3]
60 - 89 Days Past Due	127,000	[3]	74,000	[3]
Greater than 90 Days Past Due	0	[3]	0	[3]
Total Past Due	132,000	[3]	358,000	[3]
Non-accrual loans	1,283,000	[1],[3]	930,000	[3]
Total Delinquent loans	$ 1,415,000	[3]	$ 1,288,000	[3]

[1] Nonaccrual loans exclude the guaranteed portion of delinquent SBA loans that are in liquidation totaling $30.0 million.

[2] Nonaccrual loans exclude guaranteed portion of delinquent SBA loans that are in liquidation totaling $27.5 million.

[3] The Acquired Loans exclude ACILs.