Residential Whole-Loans and Bridge Loans - Aging of Delinquent Loans (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2020 USD ($) loan
Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	3,543
Principal	$ 1,352,778
Fair value	$ 1,309,795
Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	56
Principal	$ 29,650
Fair value	28,634
Financing receivable amortized cost	$ 2,600
Financing Receivables, Current | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	3,462
Principal	$ 1,313,975
Fair value	$ 1,272,791
Financing Receivables, Current | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	20
Principal	$ 13,124
Fair value	$ 13,071
Financial Asset, 1 to 29 Days Past Due | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	53
Principal	$ 22,403
Fair value	$ 21,522
Financial Asset, 1 to 29 Days Past Due | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	8
Principal	$ 4,154
Fair value	$ 4,095
Financial Asset, 30 to 59 Days Past Due | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	11
Principal	$ 5,559
Fair value	$ 5,357
Financial Asset, 30 to 59 Days Past Due | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	5
Principal	$ 2,468
Fair value	$ 2,427
Financial Asset, 60 to 89 Days Past Due | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	4
Principal	$ 3,779
Fair value	$ 3,550
Financial Asset, 60 to 89 Days Past Due | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	2
Principal	$ 973
Fair value	$ 926
Financial Asset, Equal to or Greater than 90 Days Past Due | Residential Whole-Loans
Variable Interest Entity [Line Items]
No of Loans | loan	13
Principal	$ 7,062
Fair value	$ 6,575
Financial Asset, Equal to or Greater than 90 Days Past Due | Bridge Loan
Variable Interest Entity [Line Items]
No of Loans | loan	21
Principal	$ 8,931
Fair value	$ 8,115