Loans Held for Investment (Details) (USD $)				12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011	Dec. 31, 2013 Maximum	Dec. 31, 2013 Business loans: Commercial and industrial	Dec. 31, 2012 Business loans: Commercial and industrial	Dec. 31, 2011 Business loans: Commercial and industrial	Dec. 31, 2013 Business loans: Commercial owner occupied	Dec. 31, 2012 Business loans: Commercial owner occupied	Dec. 31, 2011 Business loans: Commercial owner occupied	Dec. 31, 2013 Business loans: SBA	Dec. 31, 2012 Business loans: SBA	Dec. 31, 2011 Business loans: SBA	Dec. 31, 2013 Business loans: Warehouse facilities	Dec. 31, 2012 Business loans: Warehouse facilities	Dec. 31, 2011 Business loans: Warehouse facilities	Dec. 31, 2013 Real estate loans: Commercial non-owner occupied	Dec. 31, 2012 Real estate loans: Commercial non-owner occupied	Dec. 31, 2011 Real estate loans: Commercial non-owner occupied	Dec. 31, 2013 Real estate loans: Multi-family	Dec. 31, 2012 Real estate loans: Multi-family	Dec. 31, 2011 Real estate loans: Multi-family	Dec. 31, 2013 Real estate loans: One-to-four family	Dec. 31, 2012 Real estate loans: One-to-four family	Dec. 31, 2011 Real estate loans: One-to-four family	Dec. 31, 2013 Real estate loans: Construction	Dec. 31, 2013 Real estate loans: Land	Dec. 31, 2012 Real estate loans: Land	Dec. 31, 2011 Real estate loans: Land	Dec. 31, 2013 Other loans	Dec. 31, 2012 Other loans	Dec. 31, 2011 Other loans
Loans Held for Investment
Total gross loans	$ 1,243,252,000	$ 986,194,000			$ 187,035,000	$ 115,354,000		$ 221,089,000	$ 150,934,000		$ 10,659,000	$ 6,882,000		$ 87,517,000	$ 195,761,000		$ 333,544,000	$ 253,409,000		$ 233,689,000	$ 156,424,000		$ 145,235,000	$ 97,463,000		$ 13,040,000	$ 7,605,000	$ 8,774,000		$ 3,839,000	$ 1,193,000
Plus (less): loans held for sale, net	(3,147,000)	(3,681,000)
Plus (less): Deferred loan origination fees-net	18,000	(306,000)
Plus (less): Allowance for estimated loan losses	(8,200,000)	(7,994,000)	(8,522,000)		(1,968,000)	(1,310,000)	(1,361,000)	(1,818,000)	(1,512,000)	(1,119,000)	(151,000)	(79,000)	(80,000)	(392,000)	(1,544,000)	(1,347,000)	(1,658,000)	(1,459,000)	(1,287,000)	(817,000)	(1,145,000)	(2,281,000)	(1,099,000)	(862,000)	(931,000)	(136,000)	(127,000)	(31,000)	(39,000)	(34,000)	(52,000)	(77,000)
Loans held for investment, net	1,231,923,000	974,213,000
Secured loans limit to one borrower (as a percent)				25.00%
Unsecured loans limit to one borrower (as a percent)				15.00%
Secured loans limit to one borrower	46,800,000
Unsecured loans limit to one borrower	28,100,000
Aggregate outstanding balance of loans to one borrower of secured credit	34,700,000
Unpaid principal balance for loans and participations serviced for others	43,800,000	25,800,000
Purchased Credit Impaired Loans
Total purchase credit impaired	$ 1,977,000	$ 6,319,000	$ 6,488,000		$ 80,000	$ 308,000	$ 117,000	$ 870,000	$ 1,221,000	$ 1,930,000							$ 1,009,000	$ 2,447,000	$ 3,297,000				$ 18,000	$ 30,000	$ 1,092,000			$ 2,313,000	$ 52,000