Investment Securities (Summary of Available-for-sale and Held-to-maturity Securities) (Details) - USD ($)	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	$ 496,165,000	$ 515,250,000
AFS, debt securities, gross unrealized gains	4,598,000	305,000
AFS, debt securities, gross unrealized losses	1,768,000	12,762,000
AFS, debt securities, estimated fair value	498,995,000	502,793,000
Investment securities held-to-maturity:
HTM, amortized Cost	86,780,000	114,776,000
HTM, gross unrealized gains	935,000	32,000
HTM, gross unrealized losses	154,000	1,978,000
HTM, estimated fair value	87,561,000	112,830,000
Equity securities held-for-trading
Equity securities, gross unrealized gains	561,000
Equity securities held-for-trading	3,581,000	4,207,000
GSE debentures
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	182,922,000	184,072,000
AFS, debt securities, gross unrealized gains	2,939,000	99,000
AFS, debt securities, gross unrealized losses	58,000	3,092,000
AFS, debt securities, estimated fair value	185,803,000	181,079,000
Investment securities held-to-maturity:
HTM, amortized Cost	31,228,000	50,546,000
HTM, gross unrealized gains	113,000	22,000
HTM, gross unrealized losses	51,000	967,000
HTM, estimated fair value	31,290,000	49,601,000
GSE CMOs
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	87,001,000	107,363,000
AFS, debt securities, gross unrealized gains	22,000	17,000
AFS, debt securities, gross unrealized losses	1,091,000	4,250,000
AFS, debt securities, estimated fair value	85,932,000	103,130,000
GSE MBSs
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	153,049,000	169,334,000
AFS, debt securities, gross unrealized gains	797,000	124,000
AFS, debt securities, gross unrealized losses	503,000	4,369,000
AFS, debt securities, estimated fair value	153,343,000	165,089,000
Investment securities held-to-maturity:
HTM, amortized Cost	9,360,000	11,426,000
HTM, gross unrealized gains	0	0
HTM, gross unrealized losses	81,000	295,000
HTM, estimated fair value	9,279,000	11,131,000
SBA commercial loan asset-backed securities
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	34,000	51,000
AFS, debt securities, gross unrealized gains	0	0
AFS, debt securities, gross unrealized losses	0	0
AFS, debt securities, estimated fair value	34,000.0	51,000.0
Municipal obligations
Investment securities held-to-maturity:
HTM, amortized Cost	45,692,000	52,304,000
HTM, gross unrealized gains	822,000	10,000
HTM, gross unrealized losses	0	716,000
HTM, estimated fair value	46,514,000	51,598,000
Corporate debt obligations
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	28,484,000	40,618,000
AFS, debt securities, gross unrealized gains	502,000	0
AFS, debt securities, gross unrealized losses	0	910,000
AFS, debt securities, estimated fair value	28,986,000	39,708,000
U.S. treasury bonds
Debt Securities, Available-for-sale [Line Items]
AFS, debt securities, amortized cost	44,675,000	13,812,000
AFS, debt securities, gross unrealized gains	338,000	65,000
AFS, debt securities, gross unrealized losses	116,000	141,000
AFS, debt securities, estimated fair value	44,897,000	13,736,000
Foreign government obligations
Investment securities held-to-maturity:
HTM, amortized Cost	500,000	500,000
HTM, gross unrealized gains	0	0
HTM, gross unrealized losses	22,000	0
HTM, estimated fair value	$ 478,000	$ 500,000