Investment Securities Available For Sale - Portfolio Reset Periods (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	$ 724,720	$ 816,647
Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	102,918	126,584
6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	46,079	21,746
More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	575,723	668,317
Agency RMBS
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	713,116	779,505
Agency RMBS | Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	92,693	89,442
Agency RMBS | 6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	44,700	21,746
Agency RMBS | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	575,723	668,317
Non-Agency RMBS
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	1,567	1,939
Non-Agency RMBS | Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	188	1,939
Non-Agency RMBS | 6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	1,379	0
Non-Agency RMBS | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
U.S. Treasury securities
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	10,037	0
U.S. Treasury securities | Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	10,037	0
U.S. Treasury securities | 6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
U.S. Treasury securities | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
Collateralized Loan Obligations
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	35,203
Collateralized Loan Obligations | Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	35,203
Collateralized Loan Obligations | 6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
Collateralized Loan Obligations | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
Investment Held in Securitization Trust
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	40,734	38,594
Investment Held in Securitization Trust | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale		38,594
CMBS | Investment Held in Securitization Trust
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	40,734	38,594
CMBS | Investment Held in Securitization Trust | Less than 6 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
CMBS | Investment Held in Securitization Trust | 6 to 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	0	0
CMBS | Investment Held in Securitization Trust | More than 24 months
Schedule of Available-for-sale Securities [Line Items]
Carrying value, investment securities available for sale	$ 40,734	$ 38,594