Loans Held for Investment and Loans Held for Investment at Fair Value - Summary of Difference Between Aggregate Fair Value and Aggregate Unpaid Principal Balance by Loan Segments (Details) - Nonaccrual Status $ in Thousands	Dec. 31, 2023 USD ($)
Financing Receivable, Past Due [Line Items]
Fair Value	$ 1,306,072
Unpaid Principal Balance	1,251,395
Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	1,245,499
Unpaid Principal Balance	1,178,617
90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	60,573
Unpaid Principal Balance	72,778
Difference	(12,205)
Commercial - Purchase
Financing Receivable, Past Due [Line Items]
Fair Value	208,933
Unpaid Principal Balance	194,559
Commercial - Purchase | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	204,282
Unpaid Principal Balance	188,924
Commercial - Purchase | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	4,651
Unpaid Principal Balance	5,635
Difference	(984)
Commercial - Refinance
Financing Receivable, Past Due [Line Items]
Fair Value	237,433
Unpaid Principal Balance	219,678
Commercial - Refinance | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	230,034
Unpaid Principal Balance	210,716
Commercial - Refinance | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	7,399
Unpaid Principal Balance	8,962
Difference	(1,563)
Residential 1-4 Unit - Purchase
Financing Receivable, Past Due [Line Items]
Fair Value	251,101
Unpaid Principal Balance	246,922
Residential 1-4 Unit - Purchase | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	238,215
Unpaid Principal Balance	231,494
Residential 1-4 Unit - Purchase | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	12,886
Unpaid Principal Balance	15,428
Difference	(2,542)
Residential 1-4 Unit - Refinance
Financing Receivable, Past Due [Line Items]
Fair Value	501,950
Unpaid Principal Balance	483,899
Residential 1-4 Unit - Refinance | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	472,615
Unpaid Principal Balance	448,780
Residential 1-4 Unit - Refinance | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	29,335
Unpaid Principal Balance	35,119
Difference	(5,784)
Short Term 1-4 Unit - Purchase
Financing Receivable, Past Due [Line Items]
Fair Value	48,081
Unpaid Principal Balance	47,838
Short Term 1-4 Unit - Purchase | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	46,312
Unpaid Principal Balance	45,695
Short Term 1-4 Unit - Purchase | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	1,769
Unpaid Principal Balance	2,143
Difference	(374)
Short Term 1-4 Unit - Refinance
Financing Receivable, Past Due [Line Items]
Fair Value	58,574
Unpaid Principal Balance	58,499
Short Term 1-4 Unit - Refinance | Current to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Fair Value	54,041
Unpaid Principal Balance	53,008
Short Term 1-4 Unit - Refinance | 90+days Past Due or Nonaccrual
Financing Receivable, Past Due [Line Items]
Fair Value	4,533
Unpaid Principal Balance	5,491
Difference	$ (958)