Accrued Interest Receivable Schedule of Accounts, Notes, Loans and Financing Receivable (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	$ 85,445	$ 62,751
Commercial Real Estate Assets [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	5,740	3,699
Loans Held-for-Investment, Residential Mortgages [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	17,219	18,928
Loans Held-for-Sale, Residential Mortgages [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	159	164
Federal National Mortgage Association Certificates and Obligations (FNMA) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	42,087	25,273
Federal Home Loan Mortgage Corporation Certificates and Obligations (FHLMC) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	13,039	8,914
Government National Mortgage Association Certificates and Obligations (GNMA) [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	4,036	3,068
Mortgage-backed Securities, Issued by Private Enterprises [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	3,165	2,705
Available-for-sale Securities [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable	$ 62,327	$ 39,960