Loans Receivable and Allowance For Loan Losses - Aging Analysis of Past Due Loans (Details)	12 Months Ended
	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 4,159,000	$ 4,701,000
Current	1,347,734,000	1,321,537,000
Total loans	1,351,893,000	1,326,238,000
REAL ESTATE LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total loans	824,040,000	846,323,000
Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,985,000	3,223,000
Current	821,055,000	843,100,000
Total loans	824,040,000	846,323,000
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	210,749,000	204,699,000
Total loans	210,749,000	204,699,000
Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	533,000	0
Current	179,121,000	247,306,000
Total loans	179,654,000	247,306,000
Home equity.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	294,000	427,000
Current	37,873,000	39,831,000
Total loans	38,167,000	40,258,000
One-to-four-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,158,000	2,796,000
Current	259,381,000	246,601,000
Total loans	261,539,000	249,397,000
Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	133,931,000	104,663,000
Total loans	133,931,000	104,663,000
CONSUMER LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,174,000	1,047,000
Current	325,036,000	274,426,000
Total loans	326,210,000	275,473,000
Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	939,000	747,000
Current	209,714,000	167,046,000
Total loans	210,653,000	167,793,000
Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	127,000	146,000
Current	43,911,000	44,287,000
Total loans	44,038,000	44,433,000
Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	15,000	50,000
Current	67,164,000	57,772,000
Total loans	67,179,000	57,822,000
Other consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	93,000	104,000
Current	4,247,000	5,321,000
Total loans	4,340,000	5,425,000
COMMERCIAL BUSINESS LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	431,000
Current	201,643,000	204,011,000
Total loans	201,643,000	204,442,000
Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	431,000
Current	140,531,000	138,255,000
Total loans	140,531,000	138,686,000
Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	61,112,000	65,756,000
Total loans	61,112,000	65,756,000
30-59 Days Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,314,000	2,051,000
30-59 Days Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,536,000	1,432,000
30-59 Days Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	533,000	0
30-59 Days Past Due | Home equity.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	109,000	158,000
30-59 Days Past Due | One-to-four-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	894,000	1,274,000
30-59 Days Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | CONSUMER LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	778,000	619,000
30-59 Days Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	621,000	438,000
30-59 Days Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	71,000	62,000
30-59 Days Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	15,000	50,000
30-59 Days Past Due | Other consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	71,000	69,000
30-59 Days Past Due | COMMERCIAL BUSINESS LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	343,000	898,000
60-89 Days Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	114,000	204,000
60-89 Days Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Home equity.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due		40,000
60-89 Days Past Due | One-to-four-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	114,000	164,000
60-89 Days Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | CONSUMER LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	229,000	263,000
60-89 Days Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	187,000	196,000
60-89 Days Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	40,000	43,000
60-89 Days Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Other consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,000	24,000
60-89 Days Past Due | COMMERCIAL BUSINESS LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	431,000
60-89 Days Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	431,000
60-89 Days Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 1,502,000	1,752,000
Number of loans | loan	0
90 Days or More Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 1,335,000	1,587,000
90 Days or More Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Home equity.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	185,000	229,000
90 Days or More Past Due | One-to-four-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,150,000	1,358,000
90 Days or More Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | CONSUMER LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	167,000	165,000
90 Days or More Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	131,000	113,000
90 Days or More Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	16,000	41,000
90 Days or More Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Other consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
90 Days or more past due, still accruing interest		11,000
Total Past Due	20,000	$ 11,000
Number of loans | loan		2
90 Days or More Past Due | COMMERCIAL BUSINESS LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	$ 0
90 Days or More Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	3,033,000	3,894,000
Non-Accrual | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	2,540,000	1,781,000
Non-Accrual | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	1,086,000	0
Non-Accrual | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	0	0
Non-Accrual | Home equity.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	190,000	229,000
Non-Accrual | One-to-four-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	1,264,000	1,552,000
Non-Accrual | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	0	0
Non-Accrual | CONSUMER LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	493,000	428,000
Non-Accrual | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	451,000	367,000
Non-Accrual | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	17,000	41,000
Non-Accrual | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	0	18,000
Non-Accrual | Other consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	25,000	2,000
Non-Accrual | COMMERCIAL BUSINESS LOANS.
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	0	1,685,000
Non-Accrual | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	0	1,685,000
Non-Accrual | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, Nonaccrual Status	$ 0	$ 0