Fair Value of Financial Instruments (Fair Value Methodology) (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2018 USD ($)	Dec. 31, 2017 USD ($)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	$ 1,383,502	$ 951,811
Loan purchases excluded from Level 3 fair value methodology disclosures	282,500	373,300
Discounted cash flow
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	700,250	358,871
Liquidation model
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	683,252	592,940
Residential loans, simple average, amount	$ 400	$ 336
Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	12.20%	8.80%
Liquidation timeline (in years)	4 years 6 months	4 years 6 months
Current value of underlying properties	$ 7,950	$ 9,900
Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	(0.50%)	(8.00%)
Liquidation timeline (in years)	1 month 6 days	1 month 6 days
Current value of underlying properties	$ 2	$ 0
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.080	0.130
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.045	0.045
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.052	0.055
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.500	0.500
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.061	0.061
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.080	0.080
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.159	0.151
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.009	0.0115
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.048	0.041
Prepayment rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	3.50%	2.50%
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.241	0.065
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.041	0.029
Default rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	1 year 9 months 18 days	1 year 7 months 6 days
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	1.000	1.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.129	0.138
Loss severity | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Current value of underlying properties	$ 802	$ 772