Summary of Significant Accounting Policies (Policies)	3 Months Ended
Summary of Significant Accounting Policies (Policies)	Mar. 31, 2018
Accounting Policies [Abstract]
Basis of Presentation	Basis of Presentation and Consolidation Certain disclosures included in our Annual Report on Form 10-K are not required to be included on an interim basis in our quarterly reports on Form 10-Q. We have condensed or omitted these disclosures. Therefore, this Form 10-Q should be read in conjunction with our Annual Report on Form 10-K for the year ended December 31, 2017.
Consolidation	The condensed consolidated financial statements have been prepared in accordance with U.S. GAAP and consolidate the financial statements of the Company and our controlled subsidiaries. All significant intercompany transactions, balances, revenues and expenses are eliminated upon consolidation. In the opinion of management, the condensed consolidated financial statements reflect all adjustments, consisting of normal recurring accruals, which are necessary for a fair statement of our financial condition and results of operations for the periods presented.
Use of Estimates	Use of Estimates The preparation of condensed consolidated financial statements in conformity with U.S. GAAP requires management to make estimates and assumptions that affect the amounts reported in the condensed consolidated financial statements and accompanying notes. Examples of estimates include, but are not limited to, estimates of the fair values of financial instruments, interest income on mortgage-backed and credit risk transfer securities, allowance for loan losses and other-than-temporary impairment charges. Actual results may differ from those estimates.
Restricted Cash	Restricted Cash Restricted cash represents cash posted with the Federal Home Loan Bank of Indianapolis ("FHLBI") as collateral for secured loans and cash posted with counterparties as collateral for various derivative instruments. Cash held by counterparties as collateral is legally restricted and is not available for general corporate purposes.
Accounting Pronouncements Recently Adopted	Accounting Pronouncements Recently Adopted Effective January 1, 2018, we adopted the accounting guidance that amends certain aspects of recognition, measurement, presentation, and disclosure of financial assets and liabilities. The standard requires that all equity investments, other than those accounted for as equity method investments, be measured at fair value with changes recognized in income. As of January 1, 2018, we had three types of equity investments: investments in unconsolidated ventures, an investment in an exchange traded fund, and an investment in FHLBI stock. Our investments in unconsolidated ventures are accounted for as equity method investments, and our investment in an exchange-traded fund is measured at fair value with changes recognized in income. While the standard eliminates the cost method for equity investments without readily determinable fair values, it does allow an election to record equity investments without readily determinable fair values at cost, less impairment, and plus or minus adjustments for observable price changes. We have elected to record our investment in FHLBI stock at cost, less impairment. As such, the adoption of this accounting guidance did not impact our financial condition or results of operations. The standard also amends certain disclosure requirements for financial instruments. Refer to Note 4 - "Mortgage-Backed and Credit Risk Transfer Securities" for a tabular summary of the fair value of our available-for-sale securities and securities accounted for under the fair value option by asset type. Effective January 1, 2018, we adopted the accounting guidance that is intended to reduce diversity in how restricted cash and certain transactions are classified in the statement of cash flows. The new guidance requires that the statement of cash flows explain the difference during the period in the total of cash, cash equivalents and amounts generally described as restricted cash or restricted cash equivalents. We adopted the accounting standard on a retrospective basis, which required us to restate our statement of cash flows for the three months ended March 31, 2017. The adoption resulted in an $8.1 million decrease in net cash provided by operating activities, $10.1 million decrease in net cash used in investing activities and $2.0 million decrease in net cash provided by financing activities. We included restricted cash of $2.4 million and $620,000 as of March 31, 2018 and December 31, 2017, respectively, in our reconciliation of cash, cash equivalents and restricted cash on the condensed consolidated statements of cash flows.