Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivables Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	$ 3,174,325	$ 2,832,966
Financing Receivable, Accrued Interest, Writeoff	(1,783)	(1,145)
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	681,051	609,232
Financing Receivable, Accrued Interest, Writeoff	(132)	(115)
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	879,978	889,829
Financing Receivable, Accrued Interest, Writeoff	(517)	(253)
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	574,320	448,727
Financing Receivable, Accrued Interest, Writeoff	(297)	(219)
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	912,313	782,105
Financing Receivable, Accrued Interest, Writeoff	(717)	(298)
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	62,392	38,683
Financing Receivable, Accrued Interest, Writeoff	(24)	(22)
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	64,271	64,390
Financing Receivable, Accrued Interest, Writeoff	$ (96)	$ (238)