Loans - Investment of Originated Loans Held-for-Investment by Loan Type and Credit Quality Indicator (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	$ 2,987,067	$ 2,678,877
Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	2,968,346	2,657,356
Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	2,850	3,349
Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	15,871	18,172
Real Estate | Multifamily | Less Than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	233,251	170,832
Real Estate | Multifamily | Less Than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	232,950	170,832
Real Estate | Multifamily | Less Than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Real Estate | Multifamily | Less Than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	301	0
Real Estate | Multifamily | Equal to or Greater than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	1,966,483	1,763,114
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	1,960,984	1,756,882
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	296	613
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	5,203	5,619
Real Estate | Commercial | Less Than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	79,855	79,312
Real Estate | Commercial | Less Than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	79,485	78,917
Real Estate | Commercial | Less Than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	370	395
Real Estate | Commercial | Less Than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Real Estate | Commercial | Equal to or Greater than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	449,432	420,548
Real Estate | Commercial | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	440,065	409,155
Real Estate | Commercial | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	1,092	1,137
Real Estate | Commercial | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	8,275	10,256
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	55,060	57,065
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	52,886	54,912
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	777	747
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	1,397	1,406
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	30,295	35,368
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	29,967	34,808
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	328	560
Real Estate | Construction and Land
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	38,303	26,613
Real Estate | Construction and Land | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	38,303	26,613
Real Estate | Construction and Land | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Real Estate | Construction and Land | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Real Estate | Home Equity and Lines of Credit
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	86,848	80,315
Real Estate | Home Equity and Lines of Credit | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	86,547	80,077
Real Estate | Home Equity and Lines of Credit | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	14	27
Real Estate | Home Equity and Lines of Credit | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	287	211
Commercial and Industrial
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	45,456	44,190
Commercial and Industrial | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	45,075	43,640
Commercial and Industrial | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	301	430
Commercial and Industrial | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	80	120
Other
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	2,084	1,520
Other | Pass
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	2,084	1,520
Other | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	0	0
Other | Substandard
Financing Receivable, Recorded Investment [Line Items]
Originated loans held-for-investment, net	$ 0	$ 0