Schedule I - Financial Information (Statements of Cash Flows) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Cash flows used in operating activities	$ 451,950	$ (48,093)	$ (204,270)
Investing activities
Principal payments received on mortgage loans held for investment	92,619	114,906	162,257
Proceeds from sales of real estate owned, net	111,091	76,703	55,306
Purchases of premises and equipment	(32,866)	(27,761)	(21,573)
Decrease (increase) in restricted cash and cash equivalents	8,946	9,219	11,333
Proceeds from sale of investment	0	14,376	0
Proceeds from sale of residual interests in Residual Trusts	0	189,513	0
Capital contributions to subsidiaries and VIEs	0	0	0
Returns of capital from subsidiaries and VIEs	0	0	0
Change in due from affiliates	0	0	0
Other	(4,649)	3,511	10,948
Cash flows provided by investing activities	699,809	(454,948)	(1,244,111)
Financing activities
Payments on corporate debt	(480)	(12,901)	(17,220)
Extinguishments and settlement of debt	(31,037)	(79,877)	0
Other debt issuance costs paid	(11,039)	(13,949)	(17,281)
Repurchase of shares under stock repurchase plan	0	(28,065)	0
Change in due to affiliates	0	0	0
Other	2,189	(13,119)	(42,396)
Cash flows provided by (used in) financing activities	(1,129,989)	385,694	1,276,671
Net increase (decrease) in cash and cash equivalents	21,770	(117,347)	(171,710)
Cash and cash equivalents at the beginning of the year	202,828	320,175	491,885
Cash and cash equivalents at the end of the year	224,598	202,828	320,175
Reportable Legal Entities [Member] | Parent Company [Member]
Cash flows used in operating activities	(204,359)	(107,969)	(74,452)
Investing activities
Principal payments received on mortgage loans held for investment	940	828	535
Proceeds from sales of real estate owned, net	30	118	227
Purchases of premises and equipment	(595)	(175)	0
Decrease (increase) in restricted cash and cash equivalents	9,011	(6)	4,246
Proceeds from sale of investment	0	14,376	0
Proceeds from sale of residual interests in Residual Trusts	0	189,513	0
Capital contributions to subsidiaries and VIEs	0	(9,072)	(83,544)
Returns of capital from subsidiaries and VIEs	10,991	27,309	76,214
Change in due from affiliates	126,883	(8,331)	88,360
Other	309	2,656	(2,283)
Cash flows provided by investing activities	147,569	217,216	83,755
Financing activities
Payments on corporate debt	0	(11,250)	(15,000)
Extinguishments and settlement of debt	(31,037)	(79,877)	0
Other debt issuance costs paid	(528)	0	0
Repurchase of shares under stock repurchase plan	0	(28,065)	0
Change in due to affiliates	85,801	11,553	(98,071)
Other	(689)	(754)	6,921
Cash flows provided by (used in) financing activities	53,547	(108,393)	(106,150)
Net increase (decrease) in cash and cash equivalents	(3,243)	854	(96,847)
Cash and cash equivalents at the beginning of the year	4,016	3,162	100,009
Cash and cash equivalents at the end of the year	$ 773	$ 4,016	$ 3,162