Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivables Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019
Financing Receivable Impaired [Line Items]
Available-for-sale Securities, Amortized Cost Basis	$ 1,866,781	$ 1,685,748
Financing Receivable, Accrued Interest, Writeoff	5,717	1,330
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Available-for-sale Securities, Amortized Cost Basis	303,373	255,802
Financing Receivable, Accrued Interest, Writeoff	425	119
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Available-for-sale Securities, Amortized Cost Basis	711,397	647,838
Financing Receivable, Accrued Interest, Writeoff	2,633	558
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Available-for-sale Securities, Amortized Cost Basis	245,579	237,445
Financing Receivable, Accrued Interest, Writeoff	404	154
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Available-for-sale Securities, Amortized Cost Basis	606,432	544,663
Financing Receivable, Accrued Interest, Writeoff	$ 2,255	$ 499