Segment Information - Statements of Income (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2018	Mar. 31, 2017
Segment Information
Interest income	$ 51,612	$ 33,525
Interest expense	33,387	19,437
Net interest income	18,225	14,088
Other revenue:
Gain on sales, including fee-based services, net	18,193	19,171
Mortgage servicing rights	19,634	20,030
Servicing revenue	21,412	16,682
Amortization of MSRs	(11,865)	(11,888)
Property operating income	2,910	3,223
Other income, net	2,878	(886)
Total other revenue	53,162	46,332
Other expenses:
Employee compensation and benefits	29,494	19,841
Selling and administrative	8,915	7,695
Property operating expenses	2,796	2,638
Depreciation and amortization	1,846	1,897
Impairment loss on real estate owned		1,200
Provision for loss sharing (net of recoveries)	473	1,679
Provision for loan losses (net of recoveries)	325	(696)
Management fee - related party		4,000
Total other expenses	43,849	38,254
Income before gain on extinguishment of debt income from equity affiliates and income taxes	27,538	22,166
Gain on extinguishment of debt		7,116
Income from equity affiliates	746	763
Benefit from (provision for) income taxes	8,784	(6,101)
Net income	37,068	23,944
Preferred stock dividends	1,888	1,888
Net income attributable to noncontrolling interest	8,991	6,442
Net income attributable to common stockholders	26,189	15,614
Operating segments | Structured Business
Segment Information
Interest income	47,236	28,509
Interest expense	30,205	15,241
Net interest income	17,031	13,268
Other revenue:
Property operating income	2,910	3,223
Other income, net	233	111
Total other revenue	3,143	3,334
Other expenses:
Employee compensation and benefits	7,586	3,832
Selling and administrative	3,538	3,082
Property operating expenses	2,796	2,638
Depreciation and amortization	446	497
Impairment loss on real estate owned		1,200
Provision for loan losses (net of recoveries)	325	(696)
Management fee - related party		1,975
Total other expenses	14,691	12,528
Income before gain on extinguishment of debt income from equity affiliates and income taxes	5,483	4,074
Gain on extinguishment of debt		7,116
Income from equity affiliates	746	763
Net income	6,229	11,953
Preferred stock dividends	1,888	1,888
Net income attributable to common stockholders	4,341	10,065
Operating segments | Agency Business
Segment Information
Interest income	4,376	5,016
Interest expense	2,853	3,234
Net interest income	1,523	1,782
Other revenue:
Gain on sales, including fee-based services, net	18,193	19,171
Mortgage servicing rights	19,634	20,030
Servicing revenue	21,412	16,682
Amortization of MSRs	(11,865)	(11,888)
Other income, net	2,645	(997)
Total other revenue	50,019	42,998
Other expenses:
Employee compensation and benefits	21,908	16,009
Selling and administrative	5,377	4,613
Depreciation and amortization	1,400	1,400
Provision for loss sharing (net of recoveries)	473	1,679
Management fee - related party		2,025
Total other expenses	29,158	25,726
Income before gain on extinguishment of debt income from equity affiliates and income taxes	22,384	19,054
Benefit from (provision for) income taxes	8,784	(6,101)
Net income	31,168	12,953
Net income attributable to common stockholders	31,168	12,953
Other / Eliminations
Segment Information
Interest expense	329	962
Net interest income	(329)	(962)
Other expenses:
Income before gain on extinguishment of debt income from equity affiliates and income taxes	(329)	(962)
Net income	(329)	(962)
Net income attributable to noncontrolling interest	8,991	6,442
Net income attributable to common stockholders	$ (9,320)	$ (7,404)