Residential Whole Loans and Bridge Loans Aging of Delinquent Loans (Details)	12 Months Ended
	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)
Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	3,529
Principal	$ 1,325,443,000
Fair Value	1,375,860,000	$ 1,041,885,000
Payments of loan costs	$ 1,351,192,000	$ 1,038,008,000
Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	76
Principal	$ 37,196,000
Fair Value	36,419,000
Payments of loan costs	$ 3,100,000
Current | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	3,467
Principal	$ 1,300,238,000
Fair Value	$ 1,350,590,000
Current | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	41
Principal	$ 23,353,000
Fair Value	$ 23,329,000
1-30 days delinquent | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	41
Principal	$ 13,537,000
Fair Value	$ 14,012,000
1-30 days delinquent | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	2
Principal	$ 303,000
Fair Value	$ 306,000
31-60 days delinquent | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	5
Principal	$ 1,338,000
Fair Value	$ 1,334,000
31-60 days delinquent | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	4
Principal	$ 1,147,000
Fair Value	$ 1,135,000
61-90 days delinquent | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	4
Principal	$ 3,205,000
Fair Value	$ 3,224,000
61-90 days delinquent | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	2
Principal	$ 285,000
Fair Value	$ 280,000
90 days delinquent | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans	12
Principal	$ 7,125,000
Fair Value	$ 6,700,000
90 days delinquent | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans	27
Principal	$ 12,108,000
Fair Value	$ 11,369,000