Residential Whole Loans - Carrying Value (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest component of expected cash flows (accretable yield)	$ (28,801)	$ (3,620)	$ (98,550)	$ (34,901)
Residential whole loans held at carrying value excluded, purchase transaction not occurred	111,200		111,200
QTD
Accounts, Notes, Loans and Financing Receivable [Line Items]
Contractually required principal and interest	121,818	10,646	121,818	10,646
Contractual cash flows not expected to be collected (non-accretable yield)	(31,648)	(1,313)
Expected cash flows to be collected	90,170	9,333	90,170	9,333
Interest component of expected cash flows (accretable yield)	(28,801)	(3,620)
Fair value at the date of acquisition	61,369	5,713	61,369	5,713
YTD
Accounts, Notes, Loans and Financing Receivable [Line Items]
Contractually required principal and interest	363,144	103,950	363,144	103,950
Contractual cash flows not expected to be collected (non-accretable yield)			(66,685)	(18,465)
Expected cash flows to be collected	296,459	85,485	296,459	85,485
Interest component of expected cash flows (accretable yield)			(98,550)	(34,901)
Fair value at the date of acquisition	$ 197,909	$ 50,584	$ 197,909	$ 50,584