GiantLoop Network, Inc.
265 Winter Street . Waltham, Massachusetts 02451
October 21, 2002
Mr. Jonathan G. Katz
RE: File No. S7-32-02: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System
Dear Mr. Katz,
Please find attached the comments of GiantLoop Network, Inc. to the recent Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System, File No. S7-32-02.
GiantLoop Network is a technology solutions company based in Waltham, MA. Since the company's formation in April 2000, GiantLoop has been very active in helping large domestic financial institutions implement new data networking and storage technologies that improve companies' overall business resilience by creating secure, redundant information technology infrastructures. As such, we feel that we have a unique perspective on many of the technology issues both explicitly and implicitly raised by the sound practices proposed in the draft white paper.
We thank you for the opportunity to provide these comments. We would be happy to answer any and all questions that the agency may have regarding these comments as well as any general technology issues on which we may be able to offer a helpful perspective.
Comments filed by GiantLoop Network, Inc. to the Securities and Exchange Commission regarding File No. S7-32-02: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System
GiantLoop Network, Inc. applauds the efforts of the Securities and Exchange Commission and the other agencies involved in authoring this draft white paper (together referred to as "the agencies") in creating these guidelines for strengthening the resilience of the United States financial system.
GiantLoop is a technology solutions company based in Waltham, MA. The company offers a series of services aimed at the market for "data center networking," which the company defines as: The processing, storage, and transport of business-critical information among multiple data centers to ensure the availability, integrity, and performance of enterprise systems, applications and data. The concept of data center networking is highly relevant to the proposed sound practices because data center networking technologies are the information technology (IT) foundation upon which financial institutions will build systems and processes that will allow them to adopt the proposed guidelines for matters such as the recovery and resumption of critical activities and maintaining sufficient out-of-region resources to meet recovery and resumption objectives.
Since the company's inception, GiantLoop has been helping large U.S. financial institutions design, deploy, and manage technology solutions that help ensure the business continuity and disaster recovery readiness that is the ultimate objective of these proposed sound practices. The company counts more than twenty financial services firms among its customers, including six of the top twenty-five U.S. commercial banks and three of the top ten U.S. securities firms. With this combination of relevant technology and industry experience, GiantLoop brings an informed perspective on many of the technology issues both explicitly and implicitly raised by the sound practices proposed in the draft white paper.
While GiantLoop agrees that it is essential for the financial institutions within the scope of this white paper to establish the redundant systems and processes that will allow for the rapid recovery of those firms and their critical markets in the event of a wide-scale, regional disruption, GiantLoop believes that the agencies should adopt a practical approach in considering specific issues such as the minimum distance between a firm's primary and back-up facilities. As is described in the following comments, GiantLoop favors an empirical, company-specific approach to this issue that balances valid security and resiliency concerns with the current technological capabilities and other practical considerations of individual firms.
It should be noted that the scope of these comments are primarily limited to information technology (IT) infrastructure considerations, and the relationship between financial firms' IT resources, the current capabilities of key enabling technologies, and the agencies' proposed sound practices. GiantLoop recognizes that there are broader business process issues that will also affect financial firms ability and willingness to adopt some or all of the sound practices proposed in the draft white paper.
II. COMMENTS ON SCOPE OF APPLICATION
Regarding the agencies' request for comment on the question of the geographic concentration of back-up sites, one potential risk that the agencies might consider is that in some geographic markets, multiple companies may utilize common, third-party facilities as their back-up sites (as opposed to their own dedicated back-up site). Sound practices should take into consideration the risk that exists in certain geographies and market segments in which several companies utilize one or more shared disaster recovery sites. These risks include:
Financial firms can alleviate some of these risks by creating their own dedicated back-up infrastructure using private data networks, dedicated systems, and existing (or new) secondary facilities. GiantLoop believes that, if firms are expected to recover within the two- and four-hour timeframes proposed in the white paper, the agencies might consider specifying that a company's back-up facility(ies) are first and foremost a private facility dedicated to that firm only. In the context of the draft white paper, shared third-party back-up facilities can be best utilized as part of a multi-site back-up strategy that incorporates a shared tertiary facility - potentially hundreds of miles away - as a true "last line of defense" after dedicated primary and secondary sites that are designed to withstand - and offer rapid recovery from - wide-scale, regional disruptions.
III. COMMENTS ON RECOVERY AND RESUMPTION OF CRITICAL ACTIVITIES
From a pure information technology (IT) perspective, GiantLoop believes that the recovery and resumption targets proposed in the draft interagency white paper (two hours after an event for core clearing and settlement organizations, four hours after an event for firms that play significant roles in critical markets) are realistic and achievable given:
Financial institutions can utilize - and are utilizing - a number of computing, data storage, and data networking technologies that make the agencies' proposed recovery times possible. These technologies include:
From a technology perspective, the utilization of these technologies will make it possible for core clearing and settlement firms and other financial services companies to recover and resume critical activities within the agencies' proposed timeframes in the event of a wide-scale regional disruption. The effective utilization of these technologies, however, depends on a number of key considerations, most of which are related to the physical distance between primary and secondary data storage/processing resources. As such, these considerations are described in detail in section IV of these comments ("Comments On Sound Practices").
GiantLoop believes that the agencies should strongly consider the current capabilities of these enabling technologies as they develop their final guidelines for timeframes for recovery and resumption of critical activities. While the agencies should recognize that the final guidelines will implicitly necessitate a number of technology choices by financial firms, GiantLoop believes that the agencies should provide guidelines only for RTOs (Recovery Time Objectives - the time it takes to restore systems and operations after an outage) and RPOs (Recovery Point Objectives - the amount of acceptable data loss that results from that outage), leaving it to individual companies to determine the best set of technologies to meet those objectives given each individual company's specific IT environment.
IV. COMMENTS ON SOUND PRACTICES
In general, GiantLoop believes that the agencies have sufficiently described expectations for out-of-region back up resources. GiantLoop also believes that, to ensure the resilience of individual firms and the financial system as a whole, it is reasonable for the agencies to specify requirements for those back-up resources, i.e. that they are not dependent on the same labor pools and infrastructure components as the primary facility. However, GiantLoop believes that the agencies should not specify a minimum distance between primary sites and back-up facilities, for the following reasons:
Since there are a significant number of factors that contribute to the distance limitations of certain processing and data storage technologies (see next bullet), it is very difficult to make definitive statements as to what those distance limitations currently are. Nevertheless, some generally-accepted industry guidelines suggest that mainframe coupling (clustering) is viable up to approximately 40 km, open systems server clustering up to approximately 60 km, and synchronous data replication up to approximately 60 km or 120 km, depending on the channel protocol utilized.
The firms within the scope of the draft white paper typically have very complex IT environments that incorporate many different technologies and many different applications (many of which are internally-developed and proprietary). Since effective distance between data centers is ultimately a function of application characteristics and other technology factors, and since every firm's technology environment is unique, GiantLoop believes that specifying a standard minimum distance between primary and back-up facilities may render it, at worst impossible or, at best prohibitively complex and expensive, for many firms to meet the proposed recovery and resumption timeframes.
GiantLoop believes that the agencies should take these three factors into consideration before specifying a specific minimum distance requirement between a financial firm's primary site and back-up facility. GiantLoop's position is that the combination of these three factors makes it impractical for the agencies to specify a uniform, national minimum distance requirement, as that requirement could lead to the following adverse effects:
GiantLoop believes that the U.S. financial system will be better served if the agencies issue general guidelines that:
Given these two sets of requirements - and the technology considerations outlined above - GiantLoop believes that specific distance requirements and technology choices will be implied, and therefore, need not be explicitly specified in the final guidelines. In place of specific guidelines, GiantLoop proposes that the appropriate regulatory bodies work on a case-by-case basis with both core clearing and settlement firms and firms that play a significant role in critical financial markets to audit their IT/business resilience infrastructure.
As part of this audit, we believe that firms should be required to produce a systems proximity study. This study addresses the fact that, while there are valid limitations on the distance from a primary IT site that firms can locate a secondary facility, GiantLoop does believe that it is in the best interest of those firms and the broader U.S. financial system that companies increase the distance between those sites to either (a) the maximum effective distance their systems will allow or (b) a distance sufficient enough that the two locations are clearly not subject to the same set of natural or man-made environmental risks. In either case, the most important criteria is that the systems in question still support the recovery and resumption timeframes issued in the final guidelines. GiantLoop believes that this last point is the most critical to the rapid recovery of the financial system in the wake of a wide-scale, regional disruption, and therefore proposes that the agencies require firms to conduct technical proximity proof-of-concept studies that empirically determine the maximum effective distance of each firm's specific applications and systems ("effective" being defined as enabling the adoption of the agencies' recovery and resumption timeframes). Since many of the technologies in question are constantly evolving, these tests should be revisited on a periodic basis.
GiantLoop believes that the agencies should use the information gathered from the systems proximity study(ies) to help determine IT infrastructure requirements for each firm that will reflect both the sound practices outlined in the draft white paper as well as the unique characteristics of each firm's technology environment. GiantLoop believes that this is a practical approach that balances the interests of the regulatory agencies, the individual financial institutions, and the overall U.S. financial system.
V. COMMENTS ON TIMETABLE FOR IMPLEMENTATION
GiantLoop believes it is reasonable for the agencies to consider specific implementation timeframes, due to the importance of this issue as well as the fact that - in GiantLoop's experience - most leading financial institutions (both core clearing and settlement firms and firms that play significant roles in critical financial markets) are well underway in either planning or implementing a high-availability infrastructure for business continuity and disaster recovery purposes.
GiantLoop also believes it is reasonable for the agencies to expect that firms will achieve sound practices within the next few years. It is important, however, for the agencies to understand many of the practical IT considerations and challenges that financial firms will face as they strive to meet these new guidelines. For example, designing, sourcing, and implementing the necessary data network infrastructure to connect dispersed corporate data centers is typically a 6-12 month task. For firms that have to locate/construct, equip, and staff an entirely new data center site, 18-24 months is a more typical timeframe. Finally, many firms will have to completely re-engineer critical systems and applications in order to have them meet the more stringent recovery and resumption timeframes, an effort that requires some 18-24 months. Therefore, while there are many other business considerations that will affect adoption timeframes, GiantLoop believes that from a technology perspective, 24-36 months from the date that final guidelines are published is a reasonable balance between the importance of these guidelines and the practical aspects of adoption.
The resilience of critical U.S. financial markets in the event of wide-scale, regional disruptions is clearly of significant importance to the ongoing health of the United States and indeed the world economy. The proposed sound practices put forth by the Securities and Exchange Commission and the other agencies clearly emphasize the goal of minimizing the disruption of critical market activities in the event of such disruptions; a goal that is reflected in the aggressive recovery and resumption times proposed in the white paper.
Meeting these recovery and resumption targets can be accomplished by utilizing new computing, networking, and data storage technologies, however, the agencies should recognize that the use of these technologies does in many cases limit the maximum distance between a firm's primary and secondary data center facilities. For this reason -and the fact that different financial institutions have different technology environments and face different regional characteristics - GiantLoop believes that the agencies should not specify minimum distance requirements between primary and back-up facilities.
GiantLoop does believe that the agencies should issue guidelines specifying requirements for out-of-region back-up facilities, recovery and resumption timeframes, and should take a company-specific, empirical approach to establishing distance requirements by evaluating firms' adoption of sound practices based on the results of systems proximity studies. These tests can be used to determine the maximum distance over which a company's specific systems and applications can be operated so that they will enable the firm to meet all recovery and resumption timeframes, but not introduce increased risk to daily operations by forcing companies into unstable configurations. Given the critical nature of this issue, GiantLoop feels it is reasonable for the agencies to consider specific implementation timeframes, and that 24-36 months from the date of publication of the final guidelines is a realistic timeframe for firms to achieve sound practices.