GiantLoop Network, Inc.

265 Winter Street . Waltham, Massachusetts 02451
phone: 781-902-5100 . fax: 781.902.5200 . web: www.giantloop.com

October 21, 2002

Mr. Jonathan G. Katz
Secretary
Securities and Exchange Commission
450 5th Street, NW
Washington, DC 20549-0609

RE: File No. S7-32-02: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System

Dear Mr. Katz,

Please find attached the comments of GiantLoop Network, Inc. to the recent Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System, File No. S7-32-02.

GiantLoop Network is a technology solutions company based in Waltham, MA. Since the company's formation in April 2000, GiantLoop has been very active in helping large domestic financial institutions implement new data networking and storage technologies that improve companies' overall business resilience by creating secure, redundant information technology infrastructures. As such, we feel that we have a unique perspective on many of the technology issues both explicitly and implicitly raised by the sound practices proposed in the draft white paper.

We thank you for the opportunity to provide these comments. We would be happy to answer any and all questions that the agency may have regarding these comments as well as any general technology issues on which we may be able to offer a helpful perspective.

Sincerely,

John McKnight
Manager, Corporate Marketing
GiantLoop Network, Inc.
265 Winter Street
Waltham, MA 02451
Phone: 781-902-5184
E-mail: john.mcknight@giantloop.com

Comments filed by GiantLoop Network, Inc. to the Securities and Exchange Commission regarding File No. S7-32-02: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System

I. INTRODUCTION

GiantLoop Network, Inc. applauds the efforts of the Securities and Exchange Commission and the other agencies involved in authoring this draft white paper (together referred to as "the agencies") in creating these guidelines for strengthening the resilience of the United States financial system.

GiantLoop is a technology solutions company based in Waltham, MA. The company offers a series of services aimed at the market for "data center networking," which the company defines as: The processing, storage, and transport of business-critical information among multiple data centers to ensure the availability, integrity, and performance of enterprise systems, applications and data. The concept of data center networking is highly relevant to the proposed sound practices because data center networking technologies are the information technology (IT) foundation upon which financial institutions will build systems and processes that will allow them to adopt the proposed guidelines for matters such as the recovery and resumption of critical activities and maintaining sufficient out-of-region resources to meet recovery and resumption objectives.

Since the company's inception, GiantLoop has been helping large U.S. financial institutions design, deploy, and manage technology solutions that help ensure the business continuity and disaster recovery readiness that is the ultimate objective of these proposed sound practices. The company counts more than twenty financial services firms among its customers, including six of the top twenty-five U.S. commercial banks and three of the top ten U.S. securities firms. With this combination of relevant technology and industry experience, GiantLoop brings an informed perspective on many of the technology issues both explicitly and implicitly raised by the sound practices proposed in the draft white paper.

While GiantLoop agrees that it is essential for the financial institutions within the scope of this white paper to establish the redundant systems and processes that will allow for the rapid recovery of those firms and their critical markets in the event of a wide-scale, regional disruption, GiantLoop believes that the agencies should adopt a practical approach in considering specific issues such as the minimum distance between a firm's primary and back-up facilities. As is described in the following comments, GiantLoop favors an empirical, company-specific approach to this issue that balances valid security and resiliency concerns with the current technological capabilities and other practical considerations of individual firms.

It should be noted that the scope of these comments are primarily limited to information technology (IT) infrastructure considerations, and the relationship between financial firms' IT resources, the current capabilities of key enabling technologies, and the agencies' proposed sound practices. GiantLoop recognizes that there are broader business process issues that will also affect financial firms ability and willingness to adopt some or all of the sound practices proposed in the draft white paper.

II. COMMENTS ON SCOPE OF APPLICATION

Regarding the agencies' request for comment on the question of the geographic concentration of back-up sites, one potential risk that the agencies might consider is that in some geographic markets, multiple companies may utilize common, third-party facilities as their back-up sites (as opposed to their own dedicated back-up site). Sound practices should take into consideration the risk that exists in certain geographies and market segments in which several companies utilize one or more shared disaster recovery sites. These risks include:

  • The risk of lost systems and data - and extended operations downtime - to multiple firms in the same geography and/or market segment if a common shared back-up facility is damaged, destroyed, or rendered unavailable due to telecommunications outage, power outage, or forced employee evacuation.

  • The risk that shared back-up facilities will not help financial institutions recover and resume critical activities within the agencies' proposed recovery and resumption timeframes in the event of a wide-scale, regional disruption. Such a disruption will by definition affect many firms within a certain geographic area. If several (or all) of the firms utilizing a common shared facility claim a disaster at the same time, the ability of the shared facility to recover and restore all clients systems and data within the agencies' proposed recovery times may be brought into question.

Financial firms can alleviate some of these risks by creating their own dedicated back-up infrastructure using private data networks, dedicated systems, and existing (or new) secondary facilities. GiantLoop believes that, if firms are expected to recover within the two- and four-hour timeframes proposed in the white paper, the agencies might consider specifying that a company's back-up facility(ies) are first and foremost a private facility dedicated to that firm only. In the context of the draft white paper, shared third-party back-up facilities can be best utilized as part of a multi-site back-up strategy that incorporates a shared tertiary facility - potentially hundreds of miles away - as a true "last line of defense" after dedicated primary and secondary sites that are designed to withstand - and offer rapid recovery from - wide-scale, regional disruptions.

III. COMMENTS ON RECOVERY AND RESUMPTION OF CRITICAL ACTIVITIES

From a pure information technology (IT) perspective, GiantLoop believes that the recovery and resumption targets proposed in the draft interagency white paper (two hours after an event for core clearing and settlement organizations, four hours after an event for firms that play significant roles in critical markets) are realistic and achievable given:

  1. The current capabilities of computing, data storage, and data networking technologies, and;

  2. The deployment and utilization of those technologies in certain technical configurations and within certain geographical limits.

Enabling Technologies:

Financial institutions can utilize - and are utilizing - a number of computing, data storage, and data networking technologies that make the agencies' proposed recovery times possible. These technologies include:

  • Synchronous data replication. Synchronous data replication solutions (also often referred to as "remote mirroring" or "disk mirroring") employ sophisticated software that creates identical copies of data in multiple storage devices. These products are designed to ensure that both the primary and secondary data sets are completely synchronized, which means that companies always have an accurate, up-to-date set of production data from which to resume operations in the event of an outage at or loss of a single data center. While synchronous data replication technology deployed within a single data center offers some degree of data protection (for example, against the failure of a particular storage subsystem), for maximum availability and protection, the mirrored data sets should be in separate facilities located far enough apart that they are not subject to the same set of environmental risks (e.g. terrorist attack, major power failure, catastrophic flood, fire, etc.). This involves extending the mirroring application over a metropolitan area network (MAN) or wide area network (WAN) infrastructure. This type of remote synchronous data replication is the only practical technology that will allow firms to resume critical activities within the agencies' proposed recovery and resumption targets. Other data protection technologies (such as asynchronous data replication and tape back-up) will not provide the data currency and integrity necessary for financial firms to restore operations within the agencies' proposed recovery and resumption targets.

  • Mainframe and server clustering technologies. To ensure a truly redundant, high-availability IT infrastructure, it is not only important to have data redundancy, but also processing redundancy. Therefore, companies are deploying "Geographically Dispersed Parallel SysplexTM" (GDPS®) technology (in IBM® mainframe environments) and "server clustering" technologies (in open systems server environments) that connect multiple host computers together to essentially act as one system. This means that if one (or more) devices in a given cluster fails or is lost in an outage, that device's processing load is assumed by another device in the cluster and operations continue uninterrupted. As with data replication, the level of protection afforded by mainframe/server clustering technologies increases when the clustered devices are distributed across multiple physical sites. The most sophisticated implementations of these technologies are in so-called "active-active" data center environments. In these environments, firms employ two data centers with identical systems that equally share all processing activities at all times. This is in contrast to a primary/back-up data center environment in which the processing occurs at the primary site until a planned or unplanned outage then (and only then) forces the firm to switch all processing to the secondary data center.

  • Point-in-time copy. Among today's commercially available technologies, synchronous data replication and mainframe/server clustering technologies provide the highest levels of systems and data protection, especially if deployed in a geographically-dispersed, active-active data center environment. Point-in-time copy technologies afford firms an additional level of protection by taking and storing a "snap shot" of a set of data at a given point-in-time, which can in turn be transmitted to an out-of-region tertiary data center facility. While this point-in-time snap shot is by definition not up-to-date, it can be helpful in recovering systems and data from as close to the time of outage as possible in the event of a major wide-scale event that would disrupt or destroy both a company's primary and secondary active data centers.

From a technology perspective, the utilization of these technologies will make it possible for core clearing and settlement firms and other financial services companies to recover and resume critical activities within the agencies' proposed timeframes in the event of a wide-scale regional disruption. The effective utilization of these technologies, however, depends on a number of key considerations, most of which are related to the physical distance between primary and secondary data storage/processing resources. As such, these considerations are described in detail in section IV of these comments ("Comments On Sound Practices").

GiantLoop believes that the agencies should strongly consider the current capabilities of these enabling technologies as they develop their final guidelines for timeframes for recovery and resumption of critical activities. While the agencies should recognize that the final guidelines will implicitly necessitate a number of technology choices by financial firms, GiantLoop believes that the agencies should provide guidelines only for RTOs (Recovery Time Objectives - the time it takes to restore systems and operations after an outage) and RPOs (Recovery Point Objectives - the amount of acceptable data loss that results from that outage), leaving it to individual companies to determine the best set of technologies to meet those objectives given each individual company's specific IT environment.

IV. COMMENTS ON SOUND PRACTICES

In general, GiantLoop believes that the agencies have sufficiently described expectations for out-of-region back up resources. GiantLoop also believes that, to ensure the resilience of individual firms and the financial system as a whole, it is reasonable for the agencies to specify requirements for those back-up resources, i.e. that they are not dependent on the same labor pools and infrastructure components as the primary facility. However, GiantLoop believes that the agencies should not specify a minimum distance between primary sites and back-up facilities, for the following reasons:

  • The distance limitations of key enabling technologies. Section II, Part B, Number 3 of the draft interagency white paper states that firms ". . . should have back-up arrangements with sufficient out-of-region staff, equipment, and data to recover their critical activities within their recovery-time objectives." Assuming that those recovery-time objectives reflect the agencies' proposed recovery and resumption times (two hours after an event for core clearing and settlement organizations, four hours after an event for firms that play significant roles in critical markets), those back-up arrangements today imply the use of technologies such as synchronous data replication and server/mainframe clustering that will allow for uninterrupted access to and/or rapid restoration of data and processing resources. It is important to understand however that these technologies currently have inherent limitations that render them less effective beyond a given distance. For example, operating data replication in synchronous mode means that a write operation (from a host computer or storage device) is not considered complete until the data has been written to both the primary and secondary storage devices. As those primary and secondary devices are placed further apart, there is an increased level of latency - or the time it takes for a particular I/O (input/output) to travel from its source to its destination - introduced into the network. If these delays cross a certain threshold, data replication will terminate, and transactions will not be replicated to a secondary storage device. This means that - until replication is restored - a company will be exposed to significant data loss in the event that the primary device experiences an outage.

    Since there are a significant number of factors that contribute to the distance limitations of certain processing and data storage technologies (see next bullet), it is very difficult to make definitive statements as to what those distance limitations currently are. Nevertheless, some generally-accepted industry guidelines suggest that mainframe coupling (clustering) is viable up to approximately 40 km, open systems server clustering up to approximately 60 km, and synchronous data replication up to approximately 60 km or 120 km, depending on the channel protocol utilized.

  • Differences in firms' technology environments. It is important to recognize that the distances above are typically specifications established in the product manufacturer's test lab environment. In actual customer environments, these distance limitations can vary widely according to differences in factors such as hardware platform (e.g. open systems vs. mainframe), network media (e.g. copper vs. fiber-optics), transaction volume (i.e. higher transaction volumes typically reduce the effective distance of data replication and clustering technologies), and the characteristics of a firm's actual business applications (i.e. funds transfer, order processing, enterprise resource planning).

    The firms within the scope of the draft white paper typically have very complex IT environments that incorporate many different technologies and many different applications (many of which are internally-developed and proprietary). Since effective distance between data centers is ultimately a function of application characteristics and other technology factors, and since every firm's technology environment is unique, GiantLoop believes that specifying a standard minimum distance between primary and back-up facilities may render it, at worst impossible or, at best prohibitively complex and expensive, for many firms to meet the proposed recovery and resumption timeframes.

  • Regional differences. GiantLoop believes that in their consideration of specifying a minimum distance between primary and back-up sites, the agencies should also recognize that different geographic markets possess both different natural characteristics and different market characteristics. For example, a 50 km distance requirement between primary and back-up facilities may provide sufficient protection against all expected natural events on the Eastern seaboard, yet might leave a California-based financial institution highly vulnerable to a major earthquake if both primary and back-up sites were located on the same fault plane. From a market perspective, GiantLoop's experience in connecting corporate data centers has taught that, despite well-publicized stories in the business press about a "glut" of fiber-optic network capacity, the actual availability of that network capacity (a critical component of the technology infrastructure necessary to meet the proposed recovery and resumption guidelines) varies widely from market to market, and may hinder a firm's ability to establish timely, reliable connectivity to a remote data center.

GiantLoop believes that the agencies should take these three factors into consideration before specifying a specific minimum distance requirement between a financial firm's primary site and back-up facility. GiantLoop's position is that the combination of these three factors makes it impractical for the agencies to specify a uniform, national minimum distance requirement, as that requirement could lead to the following adverse effects:

  • An increased risk to a financial firm's daily operations due to potentially less stable geographically dispersed IT configurations.

  • An increased risk to individual financial firms and the broader U.S. financial system if these geographically dispersed IT configurations exceed the capabilities of existing technologies to enable the recovery and resumption of critical activities within the timeframes proposed in the draft white paper.

  • The adverse financial impact on the financial sector if firms must deploy long-distance back-up solutions that increase costs, but do not necessarily improve disaster protection.

GiantLoop believes that the U.S. financial system will be better served if the agencies issue general guidelines that:

  • Establish clear expectations/requirements that firms maintain "out-of-region" back-up facilities for data and operations that are not subject to the same set of risks as a firm's primary site.

  • Establish clear expectations/requirements for timeframes to recover and resume critical activities.

Given these two sets of requirements - and the technology considerations outlined above - GiantLoop believes that specific distance requirements and technology choices will be implied, and therefore, need not be explicitly specified in the final guidelines. In place of specific guidelines, GiantLoop proposes that the appropriate regulatory bodies work on a case-by-case basis with both core clearing and settlement firms and firms that play a significant role in critical financial markets to audit their IT/business resilience infrastructure.

As part of this audit, we believe that firms should be required to produce a systems proximity study. This study addresses the fact that, while there are valid limitations on the distance from a primary IT site that firms can locate a secondary facility, GiantLoop does believe that it is in the best interest of those firms and the broader U.S. financial system that companies increase the distance between those sites to either (a) the maximum effective distance their systems will allow or (b) a distance sufficient enough that the two locations are clearly not subject to the same set of natural or man-made environmental risks. In either case, the most important criteria is that the systems in question still support the recovery and resumption timeframes issued in the final guidelines. GiantLoop believes that this last point is the most critical to the rapid recovery of the financial system in the wake of a wide-scale, regional disruption, and therefore proposes that the agencies require firms to conduct technical proximity proof-of-concept studies that empirically determine the maximum effective distance of each firm's specific applications and systems ("effective" being defined as enabling the adoption of the agencies' recovery and resumption timeframes). Since many of the technologies in question are constantly evolving, these tests should be revisited on a periodic basis.

GiantLoop believes that the agencies should use the information gathered from the systems proximity study(ies) to help determine IT infrastructure requirements for each firm that will reflect both the sound practices outlined in the draft white paper as well as the unique characteristics of each firm's technology environment. GiantLoop believes that this is a practical approach that balances the interests of the regulatory agencies, the individual financial institutions, and the overall U.S. financial system.

V. COMMENTS ON TIMETABLE FOR IMPLEMENTATION

GiantLoop believes it is reasonable for the agencies to consider specific implementation timeframes, due to the importance of this issue as well as the fact that - in GiantLoop's experience - most leading financial institutions (both core clearing and settlement firms and firms that play significant roles in critical financial markets) are well underway in either planning or implementing a high-availability infrastructure for business continuity and disaster recovery purposes.

GiantLoop also believes it is reasonable for the agencies to expect that firms will achieve sound practices within the next few years. It is important, however, for the agencies to understand many of the practical IT considerations and challenges that financial firms will face as they strive to meet these new guidelines. For example, designing, sourcing, and implementing the necessary data network infrastructure to connect dispersed corporate data centers is typically a 6-12 month task. For firms that have to locate/construct, equip, and staff an entirely new data center site, 18-24 months is a more typical timeframe. Finally, many firms will have to completely re-engineer critical systems and applications in order to have them meet the more stringent recovery and resumption timeframes, an effort that requires some 18-24 months. Therefore, while there are many other business considerations that will affect adoption timeframes, GiantLoop believes that from a technology perspective, 24-36 months from the date that final guidelines are published is a reasonable balance between the importance of these guidelines and the practical aspects of adoption.

VI. SUMMARY

The resilience of critical U.S. financial markets in the event of wide-scale, regional disruptions is clearly of significant importance to the ongoing health of the United States and indeed the world economy. The proposed sound practices put forth by the Securities and Exchange Commission and the other agencies clearly emphasize the goal of minimizing the disruption of critical market activities in the event of such disruptions; a goal that is reflected in the aggressive recovery and resumption times proposed in the white paper.

Meeting these recovery and resumption targets can be accomplished by utilizing new computing, networking, and data storage technologies, however, the agencies should recognize that the use of these technologies does in many cases limit the maximum distance between a firm's primary and secondary data center facilities. For this reason -and the fact that different financial institutions have different technology environments and face different regional characteristics - GiantLoop believes that the agencies should not specify minimum distance requirements between primary and back-up facilities.

GiantLoop does believe that the agencies should issue guidelines specifying requirements for out-of-region back-up facilities, recovery and resumption timeframes, and should take a company-specific, empirical approach to establishing distance requirements by evaluating firms' adoption of sound practices based on the results of systems proximity studies. These tests can be used to determine the maximum distance over which a company's specific systems and applications can be operated so that they will enable the firm to meet all recovery and resumption timeframes, but not introduce increased risk to daily operations by forcing companies into unstable configurations. Given the critical nature of this issue, GiantLoop feels it is reasonable for the agencies to consider specific implementation timeframes, and that 24-36 months from the date of publication of the final guidelines is a realistic timeframe for firms to achieve sound practices.