Fair Value of Financial Instruments - Recorded Amount of Assets and Liabilities Measured at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	$ 540,045	$ 380,490
Mutual fund	540,045	380,490
Servicing assets	35,365	47,641	$ 52,298
US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	5,015	4,966
US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	22,779	30,944
Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	503,297	343,581
Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Servicing assets	35,365	47,641
Equity warrant assets	570	527
Total assets at fair value	578,186	430,757
Recurring | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	8,954	999
Recurring | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	5,015	4,966
Recurring | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	22,779	30,944
Recurring | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	503,297	343,581
Recurring | Mutual fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	2,206	2,099
Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Servicing assets	0	0
Equity warrant assets	0	0
Total assets at fair value	0	0
Recurring | Level 1 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | Mutual fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	0	0
Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Servicing assets	0	0
Equity warrant assets	0	0
Total assets at fair value	542,159	381,590
Recurring | Level 2 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	8,862	0
Recurring | Level 2 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	5,015	4,966
Recurring | Level 2 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	22,779	30,944
Recurring | Level 2 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	503,297	343,581
Recurring | Level 2 | Mutual fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	2,206	2,099
Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Servicing assets	35,365	47,641
Equity warrant assets	570	527
Total assets at fair value	36,027	49,167
Recurring | Level 3 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	92	999
Recurring | Level 3 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 3 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 3 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 3 | Mutual fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	$ 0	$ 0