Loans - Geographic and Collateral Concentration (Details) - Loans, held-for-investment	6 Months Ended	12 Months Ended
	Jun. 30, 2017	Dec. 31, 2016
Concentration risk
Percentage of loan	100.00%	100.00%
CALIFORNIA
Concentration risk
Percentage of loan	13.80%	12.80%
TEXAS
Concentration risk
Percentage of loan	12.70%	14.00%
FLORIDA
Concentration risk
Percentage of loan	11.20%	9.60%
NEW YORK
Concentration risk
Percentage of loan	7.00%	6.90%
ARIZONA
Concentration risk
Percentage of loan	5.90%	5.40%
GEORGIA
Concentration risk
Percentage of loan	5.60%	5.60%
NORTH CAROLINA
Concentration risk
Percentage of loan	3.10%	3.80%
ILLINOIS
Concentration risk
Percentage of loan	3.10%	2.50%
NEW JERSEY
Concentration risk
Percentage of loan	2.80%	2.90%
VIRGINIA
Concentration risk
Percentage of loan	2.60%	3.00%
Other
Concentration risk
Percentage of loan	32.20%	33.50%
Collateral concentration
Concentration risk
Percentage of loan	100.00%	100.00%
Percentage of SBA loan	100.00%	100.00%
Collateral concentration | Offices of Physicians
Concentration risk
Percentage of SBA loan	16.50%	15.80%
Collateral concentration | Child Day Care Services
Concentration risk
Percentage of SBA loan	12.20%	14.30%
Collateral concentration | Lodging
Concentration risk
Percentage of SBA loan	11.50%	11.40%
Collateral concentration | Veterinarians
Concentration risk
Percentage of SBA loan	7.00%	6.70%
Collateral concentration | Eating Places
Concentration risk
Percentage of SBA loan	5.60%	6.30%
Collateral concentration | Grocery Stores
Concentration risk
Percentage of SBA loan	4.60%	4.30%
Collateral concentration | Auto
Concentration risk
Percentage of SBA loan	3.30%	3.20%
Collateral concentration | Accounting, Auditing and Bookkeeping
Concentration risk
Percentage of SBA loan	2.50%	2.40%
Collateral concentration | Funeral Service and Crematories
Concentration risk
Percentage of SBA loan	1.90%	1.70%
Collateral concentration | Gasoline Service Stations
Concentration risk
Percentage of SBA loan	1.70%	1.80%
Collateral concentration | Other
Concentration risk
Percentage of SBA loan	33.20%	32.10%
Collateral concentration | SBA
Concentration risk
Percentage of loan	32.50%	36.00%
Collateral concentration | Retail
Concentration risk
Percentage of loan	17.60%	14.90%
Collateral concentration | Multi-family
Concentration risk
Percentage of loan	15.60%	13.30%
Collateral concentration | Office
Concentration risk
Percentage of loan	13.70%	13.50%
Collateral concentration | Industrial
Concentration risk
Percentage of loan	6.70%	6.80%
Collateral concentration | Mixed Use
Concentration risk
Percentage of loan	5.10%	5.20%
Collateral concentration | Lodging
Concentration risk
Percentage of loan	3.70%	3.70%
Collateral concentration | Other
Concentration risk
Percentage of loan	5.10%	6.60%