Business Purpose Loans - Geographic Concentration of Business Purpose Loans (Details)	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Single-family rental loans, held-for-sale at fair value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	100.00%	100.00%
Single-family rental loans, held-for-sale at fair value | Texas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	24.00%	19.00%
Single-family rental loans, held-for-sale at fair value | New Jersey
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	17.00%	12.00%
Single-family rental loans, held-for-sale at fair value | Georgia
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	12.00%	8.00%
Single-family rental loans, held-for-sale at fair value | Florida
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	10.00%	5.00%
Single-family rental loans, held-for-sale at fair value | Connecticut
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	8.00%
Single-family rental loans, held-for-sale at fair value | New York
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	5.00%
Single-family rental loans, held-for-sale at fair value | Arizona
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%
Single-family rental loans, held-for-sale at fair value | California
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	2.00%
Single-family rental loans, held-for-sale at fair value | Illinois
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	1.00%
Single-family rental loans, held-for-sale at fair value | Alabama
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	2.00%	5.00%
Single-family rental loans, held-for-sale at fair value | Indiana
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	2.00%
Single-family rental loans, held-for-sale at fair value | Tennessee
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%
Single-family rental loans, held-for-sale at fair value | Utah
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Single-family rental loans, held-for-sale at fair value | Arkansas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		6.00%
Single-family rental loans, held-for-sale at fair value | Maryland
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		6.00%
Single-family rental loans, held-for-sale at fair value | Other states (none greater than 5%)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	31.00%
Bridge loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	100.00%	100.00%
Bridge loans held-for-investment | Texas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	4.00%	3.00%
Bridge loans held-for-investment | New Jersey
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	9.00%	8.00%
Bridge loans held-for-investment | Georgia
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	8.00%	7.00%
Bridge loans held-for-investment | Florida
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	11.00%	9.00%
Bridge loans held-for-investment | Connecticut
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	1.00%
Bridge loans held-for-investment | New York
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	8.00%	6.00%
Bridge loans held-for-investment | Arizona
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	1.00%
Bridge loans held-for-investment | California
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	13.00%	21.00%
Bridge loans held-for-investment | Illinois
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	7.00%	3.00%
Bridge loans held-for-investment | Alabama
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	6.00%	4.00%
Bridge loans held-for-investment | Indiana
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%
Bridge loans held-for-investment | Tennessee
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	6.00%
Bridge loans held-for-investment | Utah
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		5.00%
Bridge loans held-for-investment | Arkansas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Bridge loans held-for-investment | Maryland
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Bridge loans held-for-investment | Other states (none greater than 5%)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	21.00%	34.00%
Redwood | Single-family rental loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	100.00%
Redwood | Single-family rental loans held-for-investment | Texas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	12.00%
Redwood | Single-family rental loans held-for-investment | New Jersey
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	5.00%
Redwood | Single-family rental loans held-for-investment | Georgia
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	0.00%
Redwood | Single-family rental loans held-for-investment | Florida
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	0.00%
Redwood | Single-family rental loans held-for-investment | Connecticut
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%
Redwood | Single-family rental loans held-for-investment | New York
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	1.00%
Redwood | Single-family rental loans held-for-investment | Arizona
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%
Redwood | Single-family rental loans held-for-investment | California
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	1.00%
Redwood | Single-family rental loans held-for-investment | Illinois
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	0.00%
Redwood | Single-family rental loans held-for-investment | Alabama
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	0.00%
Redwood | Single-family rental loans held-for-investment | Indiana
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%
Redwood | Single-family rental loans held-for-investment | Tennessee
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%
Redwood | Single-family rental loans held-for-investment | Utah
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Redwood | Single-family rental loans held-for-investment | Arkansas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Redwood | Single-family rental loans held-for-investment | Maryland
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
Redwood | Single-family rental loans held-for-investment | Other states (none greater than 5%)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	0.00%	81.00%
CAFL | Single-family rental loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	100.00%	100.00%
CAFL | Single-family rental loans held-for-investment | Texas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	15.00%	15.00%
CAFL | Single-family rental loans held-for-investment | New Jersey
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	11.00%	11.00%
CAFL | Single-family rental loans held-for-investment | Georgia
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	5.00%
CAFL | Single-family rental loans held-for-investment | Florida
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	8.00%	8.00%
CAFL | Single-family rental loans held-for-investment | Connecticut
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	4.00%
CAFL | Single-family rental loans held-for-investment | New York
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	1.00%	1.00%
CAFL | Single-family rental loans held-for-investment | Arizona
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	2.00%
CAFL | Single-family rental loans held-for-investment | California
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	5.00%	7.00%
CAFL | Single-family rental loans held-for-investment | Illinois
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	4.00%	5.00%
CAFL | Single-family rental loans held-for-investment | Alabama
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	3.00%	4.00%
CAFL | Single-family rental loans held-for-investment | Indiana
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	3.00%
CAFL | Single-family rental loans held-for-investment | Tennessee
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	3.00%
CAFL | Single-family rental loans held-for-investment | Utah
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		0.00%
CAFL | Single-family rental loans held-for-investment | Arkansas
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		2.00%
CAFL | Single-family rental loans held-for-investment | Maryland
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage		2.00%
CAFL | Single-family rental loans held-for-investment | Other states (none greater than 5%)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Concentration risk percentage	36.00%	40.00%