Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivables Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2022	Jun. 30, 2021	Jun. 30, 2022	Jun. 30, 2021
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	$ 3,123,704	$ 2,087,716	$ 3,123,704	$ 2,087,716
Financing Receivable, Accrued Interest, Writeoff	878	1,378	2,022	2,850
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	677,491	366,737	677,491	366,737
Financing Receivable, Accrued Interest, Writeoff	77	53	192	192
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	912,774	690,964	912,774	690,964
Financing Receivable, Accrued Interest, Writeoff	335	225	587	693
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	529,796	276,896	529,796	276,896
Financing Receivable, Accrued Interest, Writeoff	61	63	281	121
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	887,720	608,543	887,720	608,543
Financing Receivable, Accrued Interest, Writeoff	269	281	566	801
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	57,973	40,205	57,973	40,205
Financing Receivable, Accrued Interest, Writeoff	4	639	25	685
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	57,950	104,371	57,950	104,371
Financing Receivable, Accrued Interest, Writeoff	$ 132	$ 117	$ 371	$ 358