Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Aging Status of Amortized Cost Basis in Loans Held for Investment Portfolio (Details) - COVID-19 - USD ($)	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	$ 282,031,000
Loans, current	11,707,000
Total loans	293,738,000
Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	665,812,000	$ 286,422,000
Loans, current	1,200,969,000	1,579,163,000
Total loans	1,866,781,000	1,865,585,000
30–59 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	39,526,000
30–59 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	131,747,000	125,660,000
60–89 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	105,151,000
60–89 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	158,444,000	48,834,000
90+days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	137,354,000
90+days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	375,621,000	111,928,000
Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	19,452,000
Total loans	19,452,000
Loans Individually Evaluated | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	270,597,000
Loans, current	175,000
Total loans	270,772,000
Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	1,100,000
Total loans	1,100,000
Loans Individually Evaluated | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	20,615,000
Total loans	20,615,000
Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	10,740,000
Total loans	10,740,000
Loans Individually Evaluated | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	113,175,000
Loans, current	175,000
Total loans	113,350,000
Loans Individually Evaluated | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	22,017,000
Total loans	22,017,000
Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	7,612,000
Total loans	7,612,000
Loans Individually Evaluated | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	114,790,000
Total loans	114,790,000
Loans Individually Evaluated | 30–59 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	2,143,000
Loans Individually Evaluated | 30–59 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	8,727,000
Loans Individually Evaluated | 30–59 days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	256,000
Loans Individually Evaluated | 30–59 days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	2,143,000
Loans Individually Evaluated | 30–59 days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	6,349,000
Loans Individually Evaluated | 30–59 days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	2,122,000
Loans Individually Evaluated | 60–89 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	3,943,000
Loans Individually Evaluated | 60–89 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	10,237,000
Loans Individually Evaluated | 60–89 days | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	607,000
Loans Individually Evaluated | 60–89 days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	2,346,000
Loans Individually Evaluated | 60–89 days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	2,659,000
Loans Individually Evaluated | 60–89 days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	3,957,000
Loans Individually Evaluated | 60–89 days | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	193,000
Loans Individually Evaluated | 60–89 days | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	677,000
Loans Individually Evaluated | 60–89 days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	3,741,000
Loans Individually Evaluated | 90+days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	13,366,000
Loans Individually Evaluated | 90+days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	251,633,000
Loans Individually Evaluated | 90+days | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	493,000
Loans Individually Evaluated | 90+days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	18,013,000
Loans Individually Evaluated | 90+days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	5,938,000
Loans Individually Evaluated | 90+days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	102,869,000
Loans Individually Evaluated | 90+days | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	21,824,000
Loans Individually Evaluated | 90+days | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	6,935,000
Loans Individually Evaluated | 90+days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	108,927,000
Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	262,579,000
Loans, current	11,707,000
Total loans	274,286,000
Loans Collectively Evaluated | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	395,215,000
Loans, current	1,200,794,000
Total loans	1,596,009,000
Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	43,415,000
Loans, current	1,357,000
Total loans	44,772,000
Loans Collectively Evaluated | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	61,132,000
Loans, current	221,626,000
Total loans	282,758,000
Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	108,637,000
Loans, current	5,128,000
Total loans	113,765,000
Loans Collectively Evaluated | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	154,698,000
Loans, current	443,349,000
Total loans	598,047,000
Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	32,007,000
Loans, current	504,000
Total loans	32,511,000
Loans Collectively Evaluated | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	48,551,000
Loans, current	175,011,000
Total loans	223,562,000
Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	78,520,000
Loans, current	4,718,000
Total loans	83,238,000
Loans Collectively Evaluated | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	130,834,000
Loans, current	360,808,000
Total loans	491,642,000
Loans Collectively Evaluated | 30–59 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	37,383,000
Loans Collectively Evaluated | 30–59 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	123,020,000
Loans Collectively Evaluated | 30–59 days | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	6,397,000
Loans Collectively Evaluated | 30–59 days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	20,880,000
Loans Collectively Evaluated | 30–59 days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	14,932,000
Loans Collectively Evaluated | 30–59 days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	42,512,000
Loans Collectively Evaluated | 30–59 days | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	4,447,000
Loans Collectively Evaluated | 30–59 days | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	16,377,000
Loans Collectively Evaluated | 30–59 days | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	11,607,000
Loans Collectively Evaluated | 30–59 days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	43,251,000
Loans Collectively Evaluated | 60–89 days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	101,208,000
Loans Collectively Evaluated | 60–89 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	148,207,000
Loans Collectively Evaluated | 60–89 days | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	18,220,000
Loans Collectively Evaluated | 60–89 days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	21,454,000
Loans Collectively Evaluated | 60–89 days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	39,382,000
Loans Collectively Evaluated | 60–89 days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	57,863,000
Loans Collectively Evaluated | 60–89 days | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	12,957,000
Loans Collectively Evaluated | 60–89 days | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	17,571,000
Loans Collectively Evaluated | 60–89 days | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	30,649,000
Loans Collectively Evaluated | 60–89 days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	51,319,000
Loans Collectively Evaluated | 90+days
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	123,988,000
Loans Collectively Evaluated | 90+days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	123,988,000
Loans Collectively Evaluated | 90+days | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	18,798,000
Loans Collectively Evaluated | 90+days | Commercial - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	18,798,000
Loans Collectively Evaluated | 90+days | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	54,323,000
Loans Collectively Evaluated | 90+days | Commercial - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	54,323,000
Loans Collectively Evaluated | 90+days | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	14,603,000
Loans Collectively Evaluated | 90+days | Residential 1-4 Unit - Purchase | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	14,603,000
Loans Collectively Evaluated | 90+days | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	36,264,000
Loans Collectively Evaluated | 90+days | Residential 1-4 Unit - Refinance | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due	$ 36,264,000
Impaired loans | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		125,819,000
Loans, current		179,000
Total loans		125,998,000
Impaired loans | 30–59 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		6,195,000
Impaired loans | 60–89 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		7,696,000
Impaired loans | 90+days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		111,928,000
Nonimpaired loans | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		160,603,000
Loans, current		1,578,984,000
Total loans		1,739,587,000
Nonimpaired loans | 30–59 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		119,465,000
Nonimpaired loans | 60–89 days | Aging Status and Accrual Status
Financing Receivable Recorded Investment Past Due [Line Items]
Loans, past due		$ 41,138,000