Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivables Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2022	Dec. 31, 2021
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	$ 3,277,284	$ 2,531,826
Financing Receivable, Accrued Interest, Writeoff	(4,872)	(4,679)
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	701,408	509,421
Financing Receivable, Accrued Interest, Writeoff	(640)	(346)
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	907,097	783,260
Financing Receivable, Accrued Interest, Writeoff	(1,343)	(1,176)
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	588,433	408,770
Financing Receivable, Accrued Interest, Writeoff	(596)	(209)
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	939,305	730,321
Financing Receivable, Accrued Interest, Writeoff	(1,602)	(1,474)
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	69,884	28,989
Financing Receivable, Accrued Interest, Writeoff	(189)	(821)
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	71,157	71,065
Financing Receivable, Accrued Interest, Writeoff	$ (502)	$ (653)