Segment Information - Balance Sheet (Details) - USD ($) $ in Thousands	Mar. 31, 2018	Dec. 31, 2017
Assets:
Cash and cash equivalents	$ 102,548	$ 104,374
Restricted cash	131,659	139,398
Loans and investments, net	2,702,097	2,579,127
Loans held-for-sale, net	286,325	297,443
Capitalized mortgage servicing rights, net	255,732	252,608
Securities held-to-maturity, net	36,764	27,837
Investments in equity affiliates	23,625	23,653
Goodwill and other intangible assets	120,366	121,766
Other assets	89,652	79,739
Total assets	3,748,768	3,625,945
Liabilities:
Debt obligations	2,682,504	2,531,236
Allowance for loss-sharing obligations	31,097	30,511
Other liabilities	147,297	199,642
Total liabilities	2,860,898	2,761,389
Other / Eliminations
Liabilities:
Debt obligations		50,000
Other liabilities		1,009
Total liabilities		51,009
Structured Business | Operating segments
Assets:
Cash and cash equivalents	82,051	37,056
Restricted cash	131,264	139,398
Loans and investments, net	2,702,097	2,579,127
Investments in equity affiliates	23,625	23,653
Goodwill and other intangible assets	12,500	12,500
Other assets	72,593	66,227
Total assets	3,024,130	2,857,961
Liabilities:
Debt obligations	2,401,166	2,189,700
Other liabilities	120,095	155,814
Total liabilities	2,521,261	2,345,514
Agency Business | Operating segments
Assets:
Cash and cash equivalents	20,497	67,318
Restricted cash	395
Loans held-for-sale, net	286,325	297,443
Capitalized mortgage servicing rights, net	255,732	252,608
Securities held-to-maturity, net	36,764	27,837
Goodwill and other intangible assets	107,866	109,266
Other assets	17,059	13,512
Total assets	724,638	767,984
Liabilities:
Debt obligations	281,338	291,536
Allowance for loss-sharing obligations	31,097	30,511
Other liabilities	27,202	42,819
Total liabilities	$ 339,637	$ 364,866