Fair Value Measurements - Estimated Fair Values of the Financial Instruments (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financial assets:
Trading securities	$ 8,388	$ 11,222
Debt securities available-for-sale	1,061,443	1,138,352
Debt securities held-to-maturity	9,002	8,886
Financial liabilities:
Equity securities not been classified in the fair value	3,100	3,100
Carrying Value
Financial assets:
Cash and cash equivalents	114,699	147,818
Trading securities	8,388	11,222
Debt securities available-for-sale	1,061,443	1,138,352
Debt securities held-to-maturity	8,706	8,762
Equity securities	374	250
Federal Home Loan Bank of New York stock, at cost	29,855	39,575
Net loans held-for-investment	3,472,337	3,408,378
Derivative assets	1,072	79
Financial liabilities:
Deposits	3,485,546	3,408,233
Borrowed funds	716,357	857,004
Advance payments by borrowers for taxes and insurance	22,444	440,069
Derivative liabilities	1,075	79
Estimated Fair Value
Financial assets:
Cash and cash equivalents	114,699	147,818
Trading securities	8,388	11,222
Debt securities available-for-sale	1,061,443	1,138,352
Debt securities held-to-maturity	9,002	8,886
Equity securities	374	250
Federal Home Loan Bank of New York stock, at cost	29,855	39,575
Net loans held-for-investment	3,479,488	3,482,804
Derivative assets	1,072	79
Financial liabilities:
Deposits	3,495,318	3,412,414
Borrowed funds	733,159	862,980
Advance payments by borrowers for taxes and insurance	22,444	440,069
Derivative liabilities	1,075	79
Estimated Fair Value | Quoted Prices in Active Markets for Identical Assets (Level 1)
Financial assets:
Cash and cash equivalents	114,699	147,818
Trading securities	8,388	11,222
Debt securities available-for-sale	0	0
Debt securities held-to-maturity	0	0
Equity securities	374	250
Federal Home Loan Bank of New York stock, at cost	0	0
Net loans held-for-investment	0	0
Derivative assets	0	0
Financial liabilities:
Deposits	0	0
Borrowed funds	0	0
Advance payments by borrowers for taxes and insurance	0	0
Derivative liabilities	0	0
Estimated Fair Value | Significant Other Observable Inputs (Level 2)
Financial assets:
Cash and cash equivalents	0	0
Trading securities	0	0
Debt securities available-for-sale	1,061,443	1,138,352
Debt securities held-to-maturity	9,002	8,886
Equity securities	0	0
Federal Home Loan Bank of New York stock, at cost	29,855	39,575
Net loans held-for-investment	0	0
Derivative assets	1,072	79
Financial liabilities:
Deposits	3,495,318	3,412,414
Borrowed funds	733,159	862,980
Advance payments by borrowers for taxes and insurance	22,444	440,069
Derivative liabilities	1,075	79
Estimated Fair Value | Significant Unobservable Inputs (Level 3)
Financial assets:
Cash and cash equivalents	0	0
Trading securities	0	0
Debt securities available-for-sale	0	0
Debt securities held-to-maturity	0	0
Equity securities	0
Federal Home Loan Bank of New York stock, at cost	0	0
Net loans held-for-investment	3,479,488	3,482,804
Derivative assets	0	0
Financial liabilities:
Deposits	0	0
Borrowed funds	0	0
Advance payments by borrowers for taxes and insurance	0	0
Derivative liabilities	$ 0	$ 0