Accrued Interest Receivable Schedule of Accounts, Notes, Loans and Financing Receivable (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	$ 63,915	$ 49,970
Loans Held-for-sale, Residential Mortgages [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	2,724	4,173
Loans Held-for-investment, Residential Mortgages [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	19,453	18,339
Commercial Real Estate Assets [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	2,162	1,567
Federal Home Loan Mortgage Corporation Certificates and Obligations (FHLMC) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	5,696	6,235
Federal National Mortgage Association Certificates and Obligations (FNMA) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	28,587	12,407
Government National Mortgage Association Certificates and Obligations (GNMA) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	3,114	4,910
Mortgage-backed Securities, Issued by Private Enterprises [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	2,179	2,339
Available-for-sale Securities [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Receivable	$ 39,576	$ 25,891