Loans - Investment of Originated Loans Held-for-Investment by Loan Type and Credit Quality Indicator (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	$ 3,098,760	$ 2,987,067
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	3,083,462	2,968,346
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	1,699	2,850
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	13,599	15,871
Real Estate | Multifamily | Less Than 35% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	247,893	233,251
Real Estate | Multifamily | Less Than 35% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	247,304	232,950
Real Estate | Multifamily | Less Than 35% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	291	0
Real Estate | Multifamily | Less Than 35% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	298	301
Real Estate | Multifamily | Equal to or Greater than 35% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	2,056,494	1,966,483
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	2,051,039	1,960,984
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	292	296
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	5,163	5,203
Real Estate | Commercial mortgage | Less Than 35% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	89,069	79,855
Real Estate | Commercial mortgage | Less Than 35% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	89,069	79,485
Real Estate | Commercial mortgage | Less Than 35% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	370
Real Estate | Commercial mortgage | Less Than 35% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	0
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	439,035	449,432
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	432,711	440,065
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	1,092
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	6,324	8,275
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	55,424	55,060
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	53,091	52,886
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	770	777
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	1,563	1,397
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	30,980	30,295
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	30,953	29,967
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	0
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	27	328
Real Estate | Construction and land
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	39,953	38,303
Real Estate | Construction and land | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	39,953	38,303
Real Estate | Construction and land | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	0
Real Estate | Construction and land | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	0
Real Estate | Home equity and lines of credit
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	89,349	86,848
Real Estate | Home equity and lines of credit | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	89,082	86,547
Real Estate | Home equity and lines of credit | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	80	14
Real Estate | Home equity and lines of credit | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	187	287
Commercial and Industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	49,358	45,456
Commercial and Industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	49,055	45,075
Commercial and Industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	266	301
Commercial and Industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	37	80
Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	1,205	2,084
Other | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	1,205	2,084
Other | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	0	0
Other | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable	$ 0	$ 0