Variable Interest Entities Investment Portfolio (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($) item loan	Dec. 31, 2016 USD ($) item loan
Residential Portfolio Segment Current Coupon Rate One
Variable Interest Entity [Line Items]
Number of Loans | loan	142	59
Principal balance | $	$ 55,593	$ 23,318
Original LTV	55.50%	54.80%
Original FICO Score | item	751	732
Expected Life (in years)	1 year 8 months 12 days	1 year 4 months 24 days
Contractual maturity (in years)	29 years 1 month 6 days	26 years 6 months
Interest Rate	3.90%	4.20%
Residential Portfolio Segment Current Coupon Rate Two
Variable Interest Entity [Line Items]
Number of Loans | loan	338	180
Principal balance | $	$ 125,860	$ 69,930
Original LTV	56.90%	57.10%
Original FICO Score | item	725	728
Expected Life (in years)	1 year 4 months 24 days	1 year 6 months
Contractual maturity (in years)	26 years 6 months	27 years 3 months 18 days
Interest Rate	4.50%	4.60%
Residential Portfolio Segment Current Coupon Rate Three
Variable Interest Entity [Line Items]
Number of Loans | loan	132	231
Principal balance | $	$ 48,553	$ 91,440
Original LTV	58.20%	55.50%
Original FICO Score | item	728	723
Expected Life (in years)	1 year 7 months 6 days	1 year 7 months 6 days
Contractual maturity (in years)	27 years	27 years 1 month 6 days
Interest Rate	5.20%	5.00%
Residential Portfolio Segment Current Coupon Rate Four
Variable Interest Entity [Line Items]
Number of Loans | loan	4	5
Principal balance | $	$ 2,264	$ 3,077
Original LTV	71.10%	71.20%
Original FICO Score | item	758	738
Expected Life (in years)	1 year 3 months 18 days	1 year 3 months 18 days
Contractual maturity (in years)	20 years 6 months	21 years 1 month 6 days
Interest Rate	6.30%	6.30%
Residential Bridge Loan Current Coupon Rate One
Variable Interest Entity [Line Items]
Number of Loans | loan	9
Principal balance | $	$ 4,016
Original LTV	64.50%
Contractual maturity (in years)	10 months 24 days
Interest Rate	5.90%
Residential Bridge Loan Current Coupon Rate Two
Variable Interest Entity [Line Items]
Number of Loans | loan	64
Principal balance | $	$ 18,420
Original LTV	67.80%
Contractual maturity (in years)	10 months 17 days
Interest Rate	6.70%
Residential Bridge Loan Current Coupon Rate Three
Variable Interest Entity [Line Items]
Number of Loans | loan	98
Principal balance | $	$ 25,608
Original LTV	66.40%
Contractual maturity (in years)	9 months 15 days
Interest Rate	7.60%
Residential Bridge Loan Current Coupon Rate Four
Variable Interest Entity [Line Items]
Number of Loans | loan	56
Principal balance | $	$ 19,728
Original LTV	70.30%
Contractual maturity (in years)	11 months 27 days
Interest Rate	8.90%
Residential Bridge Loan Current Coupon Rate Five
Variable Interest Entity [Line Items]
Number of Loans | loan	67
Principal balance | $	$ 25,001
Original LTV	73.30%
Contractual maturity (in years)	6 months 24 days
Interest Rate	9.70%
Residential Bridge Loan Current Coupon Rate Six
Variable Interest Entity [Line Items]
Number of Loans | loan	36
Principal balance | $	$ 10,656
Original LTV	75.40%
Contractual maturity (in years)	5 months
Interest Rate	10.80%
Residential Bridge Loan Current Coupon Rate Seven
Variable Interest Entity [Line Items]
Number of Loans | loan	2
Principal balance | $	$ 919
Original LTV	89.80%
Contractual maturity (in years)	8 months 6 days
Interest Rate	11.40%
Residential Bridge Loan Current Coupon Rate Eight
Variable Interest Entity [Line Items]
Number of Loans | loan	8
Principal balance | $	$ 1,520
Original LTV	73.80%
Contractual maturity (in years)	5 months 27 days
Interest Rate	18.00%
Residential Whole-Loans
Variable Interest Entity [Line Items]
Number of Loans | loan	616	475
Principal balance | $	$ 232,270	$ 187,765
Original LTV	57.00%	56.30%
Original FICO Score | item	734	726
Expected Life (in years)	1 year 6 months	1 year 6 months
Contractual maturity (in years)	27 years 1 month 6 days	27 years
Interest Rate	4.50%	4.80%
Residential Portfolio Segment With No Fico Score
Variable Interest Entity [Line Items]
Number of Loans | loan	141	153
Principal balance | $	$ 56,500	$ 66,700
VIE | Residential Bridge Loans
Variable Interest Entity [Line Items]
Number of Loans | loan	340
Principal balance | $	$ 105,868
Original LTV	70.10%
Contractual maturity (in years)	9 months
Interest Rate	8.60%
Maximum | Residential Portfolio Segment Current Coupon Rate One
Variable Interest Entity [Line Items]
Interest Rate	4.00%	4.00%
Maximum | Residential Portfolio Segment Current Coupon Rate Two
Variable Interest Entity [Line Items]
Interest Rate	5.00%	5.00%
Maximum | Residential Portfolio Segment Current Coupon Rate Three
Variable Interest Entity [Line Items]
Interest Rate	6.00%	6.00%
Maximum | Residential Portfolio Segment Current Coupon Rate Four
Variable Interest Entity [Line Items]
Interest Rate	7.00%	7.00%
Maximum | Residential Bridge Loan Current Coupon Rate One
Variable Interest Entity [Line Items]
Interest Rate	6.00%
Maximum | Residential Bridge Loan Current Coupon Rate Two
Variable Interest Entity [Line Items]
Interest Rate	7.00%
Maximum | Residential Bridge Loan Current Coupon Rate Three
Variable Interest Entity [Line Items]
Interest Rate	8.00%
Maximum | Residential Bridge Loan Current Coupon Rate Four
Variable Interest Entity [Line Items]
Interest Rate	9.00%
Maximum | Residential Bridge Loan Current Coupon Rate Five
Variable Interest Entity [Line Items]
Interest Rate	10.00%
Maximum | Residential Bridge Loan Current Coupon Rate Six
Variable Interest Entity [Line Items]
Interest Rate	11.00%
Maximum | Residential Bridge Loan Current Coupon Rate Seven
Variable Interest Entity [Line Items]
Interest Rate	12.00%
Maximum | Residential Bridge Loan Current Coupon Rate Eight
Variable Interest Entity [Line Items]
Interest Rate	18.00%
Minimum | Residential Portfolio Segment Current Coupon Rate One
Variable Interest Entity [Line Items]
Interest Rate	3.01%	3.01%
Minimum | Residential Portfolio Segment Current Coupon Rate Two
Variable Interest Entity [Line Items]
Interest Rate	4.01%	4.01%
Minimum | Residential Portfolio Segment Current Coupon Rate Three
Variable Interest Entity [Line Items]
Interest Rate	5.01%	5.01%
Minimum | Residential Portfolio Segment Current Coupon Rate Four
Variable Interest Entity [Line Items]
Interest Rate	6.01%	6.01%
Minimum | Residential Bridge Loan Current Coupon Rate One
Variable Interest Entity [Line Items]
Interest Rate	5.01%
Minimum | Residential Bridge Loan Current Coupon Rate Two
Variable Interest Entity [Line Items]
Interest Rate	6.01%
Minimum | Residential Bridge Loan Current Coupon Rate Three
Variable Interest Entity [Line Items]
Interest Rate	7.01%
Minimum | Residential Bridge Loan Current Coupon Rate Four
Variable Interest Entity [Line Items]
Interest Rate	8.01%
Minimum | Residential Bridge Loan Current Coupon Rate Five
Variable Interest Entity [Line Items]
Interest Rate	9.01%
Minimum | Residential Bridge Loan Current Coupon Rate Six
Variable Interest Entity [Line Items]
Interest Rate	10.01%
Minimum | Residential Bridge Loan Current Coupon Rate Seven
Variable Interest Entity [Line Items]
Interest Rate	11.01%
Minimum | Residential Bridge Loan Current Coupon Rate Eight
Variable Interest Entity [Line Items]
Interest Rate	17.01%