Fair Value Measurement - Estimated Fair Values of The Financial Instruments (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financial assets:
Debt securities available-for-sale	$ 1,138,352	$ 808,031
Debt securities held-to-maturity	8,886	9,249
Financial liabilities:
Investments measured at net asset value	3,100	1,000
Carrying Value
Financial assets:
Cash and cash equivalents	147,818	77,762
Trading securities	11,222	8,968
Debt securities available-for-sale	1,138,352	808,031
Debt securities held-to-maturity	8,762	9,505
Equity securities	250	237
FHLB of New York stock, at cost	39,575	22,517
Net loans held-for-investment	3,408,378	3,217,673
Financial liabilities:
Deposits	3,408,233	3,286,512
Repurchase agreements, FHLB advances and other borrowings	857,004	408,891
Advance payments by borrowers	44,069	18,007
Estimated Fair Value
Financial assets:
Cash and cash equivalents	147,818	77,762
Trading securities	11,222	8,968
Debt securities available-for-sale	1,138,352	808,031
Debt securities held-to-maturity	8,886	9,249
Equity securities	250	237
FHLB of New York stock, at cost	39,575	22,517
Net loans held-for-investment	3,482,804	3,236,136
Financial liabilities:
Deposits	3,412,414	3,291,085
Repurchase agreements, FHLB advances and other borrowings	862,980	403,476
Advance payments by borrowers	44,069	18,007
Estimated Fair Value | Level 1
Financial assets:
Cash and cash equivalents	147,818	77,762
Trading securities	11,222	8,968
Debt securities available-for-sale	0	0
Debt securities held-to-maturity	0	0
Equity securities	250	237
FHLB of New York stock, at cost	0	0
Net loans held-for-investment	0	0
Financial liabilities:
Deposits	0	0
Repurchase agreements, FHLB advances and other borrowings	0	0
Advance payments by borrowers	0	0
Estimated Fair Value | Level 2
Financial assets:
Cash and cash equivalents	0	0
Trading securities	0	0
Debt securities available-for-sale	1,138,352	808,031
Debt securities held-to-maturity	8,886	9,249
Equity securities	0	0
FHLB of New York stock, at cost	39,575	22,517
Net loans held-for-investment	0	0
Financial liabilities:
Deposits	3,412,414	3,291,085
Repurchase agreements, FHLB advances and other borrowings	862,980	403,476
Advance payments by borrowers	44,069	18,007
Estimated Fair Value | Level 3
Financial assets:
Cash and cash equivalents	0	0
Trading securities	0	0
Debt securities available-for-sale	0	0
Debt securities held-to-maturity	0	0
Equity securities	0	0
FHLB of New York stock, at cost	0	0
Net loans held-for-investment	3,482,804	3,236,136
Financial liabilities:
Deposits	0	0
Repurchase agreements, FHLB advances and other borrowings	0	0
Advance payments by borrowers	$ 0	$ 0