Residential Loans - Geographic Concentration of Residential Loans Recorded on Consolidated Balance Sheet (Details)	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015
Residential Loans Held For Sale
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Residential Loans Held For Sale | California
Concentration Risk [Line Items]
Concentration risk percentage	40.00%	41.00%
Residential Loans Held For Sale | Texas
Concentration Risk [Line Items]
Concentration risk percentage	9.00%	9.00%
Residential Loans Held For Sale | Washington
Concentration Risk [Line Items]
Concentration risk percentage	8.00%	6.00%
Residential Loans Held For Sale | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	5.00%
Residential Loans Held For Sale | Florida
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	4.00%
Residential Loans Held For Sale | Virginia
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	3.00%
Residential Loans Held For Sale | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	3.00%
Residential Loans Held For Sale | Massachusetts
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Residential Loans Held For Sale | New York
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	1.00%
Residential Loans Held For Sale | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	28.00%	26.00%
Residential loans, held-for-investment
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Residential loans, held-for-investment | California
Concentration Risk [Line Items]
Concentration risk percentage	18.00%	18.00%
Residential loans, held-for-investment | Texas
Concentration Risk [Line Items]
Concentration risk percentage	6.00%	6.00%
Residential loans, held-for-investment | Washington
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Residential loans, held-for-investment | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	3.00%
Residential loans, held-for-investment | Florida
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	14.00%
Residential loans, held-for-investment | Virginia
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	3.00%
Residential loans, held-for-investment | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	5.00%
Residential loans, held-for-investment | Massachusetts
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Residential loans, held-for-investment | New York
Concentration Risk [Line Items]
Concentration risk percentage	8.00%	8.00%
Residential loans, held-for-investment | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	39.00%	39.00%
Residential Loans Held For Investment at Fair Value
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Residential Loans Held For Investment at Fair Value | California
Concentration Risk [Line Items]
Concentration risk percentage	42.00%	39.00%
Residential Loans Held For Investment at Fair Value | Texas
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	11.00%
Residential Loans Held For Investment at Fair Value | Washington
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	3.00%
Residential Loans Held For Investment at Fair Value | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	5.00%
Residential Loans Held For Investment at Fair Value | Florida
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	4.00%
Residential Loans Held For Investment at Fair Value | Virginia
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	4.00%
Residential Loans Held For Investment at Fair Value | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	1.00%
Residential Loans Held For Investment at Fair Value | Massachusetts
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	4.00%
Residential Loans Held For Investment at Fair Value | New York
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	5.00%
Residential Loans Held For Investment at Fair Value | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	23.00%	24.00%