Investment Securities Available For Sale - Available-For-Sale Securities (Details) - USD ($) $ in Thousands		Dec. 31, 2015	Dec. 31, 2014
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		$ 756,306	$ 836,537
Unrealized Gains		1,901	10,592
Unrealized Losses		(33,487)	(30,482)
Investment securities available for sale		724,720	816,647
Agency ARMs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		175,160	183,341
Unrealized Gains		217	136
Unrealized Losses		(2,364)	(1,784)
Investment securities available for sale		173,013	181,693
Agency Fixed Rate
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		437,683	489,494
Unrealized Gains		9	0
Unrealized Losses		(12,979)	(10,813)
Investment securities available for sale		424,713	478,681
Agency IOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs	[1]	131,623	135,501
Unrealized Gains	[1]	1,624	1,324
Unrealized Losses	[1]	(17,857)	(17,694)
Investment securities available for sale	[1]	115,390	119,131
Agency RMBS
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		744,466	808,336
Unrealized Gains		1,850	1,460
Unrealized Losses		(33,200)	(30,291)
Investment securities available for sale		713,116	779,505
Non-Agency RMBS
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		1,727	2,061
Unrealized Gains		51	69
Unrealized Losses		(211)	(191)
Investment securities available for sale		1,567	1,939
U.S. Treasury securities
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		10,113	0
Unrealized Gains		0	0
Unrealized Losses		(76)	0
Investment securities available for sale		10,037	0
Collateralized Loan Obligation
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		0	26,140
Unrealized Gains		0	9,063
Unrealized Losses		0	0
Investment securities available for sale		0	35,203
Investment Held in Securitization Trust
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		28,692	26,193
Unrealized Gains		12,042	12,401
Unrealized Losses		0	0
Investment securities available for sale		40,734	38,594
Freddie Mac | Agency ARMs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		62,383	57,597
Unrealized Gains		41	55
Unrealized Losses		(770)	(488)
Investment securities available for sale		61,654	57,164
Freddie Mac | Agency Fixed Rate
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		31,076	37,800
Unrealized Gains		0	0
Unrealized Losses		(719)	(629)
Investment securities available for sale		30,357	37,171
Freddie Mac | Agency IOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs	[1]	28,970	38,844
Unrealized Gains	[1]	680	273
Unrealized Losses	[1]	(4,471)	(5,779)
Investment securities available for sale	[1]	25,179	33,338
Fannie Mae | Agency ARMs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		92,605	113,192
Unrealized Gains		121	81
Unrealized Losses		(1,334)	(1,037)
Investment securities available for sale		91,392	112,236
Fannie Mae | Agency Fixed Rate
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		380,684	451,694
Unrealized Gains		0	0
Unrealized Losses		(12,149)	(10,184)
Investment securities available for sale		368,535	441,510
Fannie Mae | Agency IOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs	[1]	39,603	53,666
Unrealized Gains	[1]	433	741
Unrealized Losses	[1]	(6,341)	(6,388)
Investment securities available for sale	[1]	33,695	48,019
Ginnie Mae | Agency ARMs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		20,172	12,552
Unrealized Gains		55	0
Unrealized Losses		(260)	(259)
Investment securities available for sale		19,967	12,293
Ginnie Mae | Agency Fixed Rate
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		25,923	0
Unrealized Gains		9	0
Unrealized Losses		(111)	0
Investment securities available for sale		25,821	0
Ginnie Mae | Agency IOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs	[1]	63,050	42,991
Unrealized Gains	[1]	511	310
Unrealized Losses	[1]	(7,045)	(5,527)
Investment securities available for sale	[1]	56,516	37,774
CMBS | Investment Held in Securitization Trust
Schedule of Available-for-sale Securities [Line Items]
Amortized Costs		28,692	26,193
Unrealized Gains		12,042	12,401
Unrealized Losses		0	0
Investment securities available for sale		$ 40,734	$ 38,594

[1] Included in investment securities available for sale as of December 31, 2015 and 2014 are Agency IOs. Agency IOs are measured at fair value through earnings.