Principles of Consolidation - Assets and Liabilities of Consolidated Variable Interest Entity's (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($) Investment	Dec. 31, 2015 USD ($) Investment
Variable Interest Entity [Line Items]
Assets	$ 798,317	$ 1,195,574
Liabilities	$ 773,980	$ 1,050,861
Number of VIEs | Investment	20	22
Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 791,636	$ 1,021,870
Commercial loans, held-for-investment
Variable Interest Entity [Line Items]
Assets		166,016
Restricted cash
Variable Interest Entity [Line Items]
Assets	148	365
Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	1,000	2,428
Other assets
Variable Interest Entity [Line Items]
Assets	5,533	4,895
Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	518	804
Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities		100
Asset-backed securities issued, net
Variable Interest Entity [Line Items]
Liabilities	773,462	1,049,957
Sequoia Entities
Variable Interest Entity [Line Items]
Assets	798,317	1,028,124
Liabilities	$ 773,980	$ 997,475
Number of VIEs | Investment	20	21
Sequoia Entities | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 791,636	$ 1,021,870
Sequoia Entities | Commercial loans, held-for-investment
Variable Interest Entity [Line Items]
Assets		0
Sequoia Entities | Restricted cash
Variable Interest Entity [Line Items]
Assets	148	228
Sequoia Entities | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	1,000	1,131
Sequoia Entities | Other assets
Variable Interest Entity [Line Items]
Assets	5,533	4,895
Sequoia Entities | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	518	555
Sequoia Entities | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities		100
Sequoia Entities | Asset-backed securities issued, net
Variable Interest Entity [Line Items]
Liabilities	773,462	996,820
Commercial Securitization
Variable Interest Entity [Line Items]
Assets	0	167,450
Liabilities	$ 0	$ 53,386
Number of VIEs | Investment	0	1
Commercial Securitization | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 0	$ 0
Commercial Securitization | Commercial loans, held-for-investment
Variable Interest Entity [Line Items]
Assets		166,016
Commercial Securitization | Restricted cash
Variable Interest Entity [Line Items]
Assets	0	137
Commercial Securitization | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	0	1,297
Commercial Securitization | Other assets
Variable Interest Entity [Line Items]
Assets	0	0
Commercial Securitization | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	0	249
Commercial Securitization | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities		0
Commercial Securitization | Asset-backed securities issued, net
Variable Interest Entity [Line Items]
Liabilities	$ 0	$ 53,137