Recently Issued Accounting Pronouncements - Schedule of Impact of Adoption (Details) - USD ($)	Jun. 30, 2020	Mar. 31, 2020	Jan. 01, 2020	Dec. 31, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, Held-to-maturity			$ 0
Allowance for credit losses on debt securities, Available-for-sale			0
Allowance for credit losses on loans	$ 282,271,000	$ 115,422,000		$ 35,698,000	$ 35,026,000	$ 37,856,000	$ 36,072,000
Deferred tax (liabilities) assets			(1,371,000)
Allowance for credit losses on off-balance sheet credit exposures	22,000,000.0		3,279,000	3,300,000
Retained earnings	$ 247,078,000		396,051,000	396,051,000
Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			9,027,000
Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			5,492,000
Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			20,118,000
Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			1,061,000
Impact of CECL Adoption
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, Held-to-maturity			0
Allowance for credit losses on debt securities, Available-for-sale			0
Allowance for credit losses on loans			64,000,000.0	$ 55,686,000
Deferred tax (liabilities) assets			18,346,000
Allowance for credit losses on off-balance sheet credit exposures			8,285,000
Retained earnings			(45,625,000)
Impact of CECL Adoption | Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			16,072,000
Impact of CECL Adoption | Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			27,572,000
Impact of CECL Adoption | Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			9,519,000
Impact of CECL Adoption | Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			2,523,000
As Reported Under CECL
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, Held-to-maturity			0
Allowance for credit losses on debt securities, Available-for-sale			0
Deferred tax (liabilities) assets			16,975,000
Allowance for credit losses on off-balance sheet credit exposures			11,564,000
Retained earnings			350,426,000
As Reported Under CECL | Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			25,099,000
As Reported Under CECL | Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			33,064,000
As Reported Under CECL | Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			29,637,000
As Reported Under CECL | Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans			$ 3,584,000