Loans Held for Investment - Narrative (Details)	3 Months Ended	12 Months Ended
	Mar. 31, 2020 USD ($) area loan grade	Dec. 31, 2019 USD ($) loan
Loans Held for Investment
Servicing rights retained from guaranteed portion of SBA loans sold	$ 7,300,000	$ 7,700,000
Unpaid principal balance for loans and participations serviced for others	614,700,000	633,800,000
Secured loans limit to one borrower	581,500,000
Unsecured loans limit to one borrower	348,900,000
Aggregate outstanding balance of loans to one borrower of secured credit	$ 128,400,000
Number of areas where the entity's credit quality is maintained and credit risk managed | area	2
Number of Pass scale grades | grade	6
Loans Evaluated Individually for Impairment	$ 22,300,000	22,842,000
Specific Allowance for Impaired Loans	1,900,000	0
Loans on nonaccrual status	20,610,000	8,500,000
Loans 90 days or more past due and still accruing		0
Amortized cost of TDRs	$ 2,300,000	$ 3,000,000.0
Number of TDR loans | loan	2	2
TDRs to become nonaccrual | loan	1
Consumer mortgage loans collateralized by residential real estate, foreclosure proceedings in process	$ 0	$ 0
Purchased credit impaired loans, nonaccrual status	0	0
Outstanding balance		1,200,000
Discounted cash flow approach
Loans Held for Investment
Loans Evaluated Individually for Impairment	16,300,000
Underlying value of the collateral
Loans Held for Investment
Loans Evaluated Individually for Impairment	6,000,000.0
Secured Debt
Loans Held for Investment
Aggregate outstanding balance of loans to one borrower of secured credit	101,500,000
Unsecured Debt
Loans Held for Investment
Aggregate outstanding balance of loans to one borrower of secured credit	26,900,000
SBA
Loans Held for Investment
Unpaid principal balance for loans and participations serviced for others	$ 455,900,000	$ 475,300,000