Residential Whole Loans (Purchased Performing Loans) (Details)	3 Months Ended	6 Months Ended
	Jun. 30, 2019 USD ($) loan	Jun. 30, 2019 USD ($) loan	Jun. 30, 2018 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average LTV Ratio		80.00%
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Provisions/(reversal of provisions) for loan losses	$ 385,000	$ 568,000
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of other loans at carrying value | loan	98	98
Unpaid balance of other loans	$ 38,300,000	$ 38,300,000
Total outstanding principal balance of all of the Company’s purchased performing loans	1.10%	1.10%
Weighted Average LTV Ratio		67.00%
Allowance for loan loss	$ 450,000	$ 450,000
Provisions/(reversal of provisions) for loan losses	0	622,000	$ 0
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Commitment to lend, unfunded	$ 100,400,000	$ 100,400,000