Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivable Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	$ 2,832,892	$ 3,277,284
Financing Receivable, Accrued Interest, Writeoff	(7,662)	(4,872)
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	631,925	701,408
Financing Receivable, Accrued Interest, Writeoff	(788)	(640)
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	799,816	907,097
Financing Receivable, Accrued Interest, Writeoff	(2,084)	(1,343)
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	511,080	588,433
Financing Receivable, Accrued Interest, Writeoff	(1,239)	(596)
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	808,149	939,305
Financing Receivable, Accrued Interest, Writeoff	(2,918)	(1,602)
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	41,111	69,884
Financing Receivable, Accrued Interest, Writeoff	(174)	(189)
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	40,811	71,157
Financing Receivable, Accrued Interest, Writeoff	$ (459)	$ (502)