Endur, Inc.

To: Ms. Jennifer J. Johnson
Secretary
Board of Governors of the Federal Reserve System
20th Street and Constitution Avenue, NW
Washington, D.C. 20551
Attention: Docket No. R-1128

Office of the Comptroller of the Currency
250 E Street, SW
Public Information Room, Mail Stop 1-5
Washington, DC 20219
Attention: Docket No. 02-13

Jonathan G. Katz
Secretary
Securities and Exchange Commission
450 5th Street, NW
Washington, DC 20549-0609
Attention: File No S7-32-02

From: Chris Kay
VP, Campus Development
Endur
216 16th Street, Suite 730
Denver, CO 80202
ckay@endurcorp.com
Date: October 21st, 2002
Subject: Comments on Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U. S. Financial System

Attached are Endur's comments on the draft interagency white paper. As a provider of highly reliable, highly secure mission critical facilities, our emphasis is on the placement, design, development, and operation of these facilities. We thank you for the opportunity to comment on the practices identified in the white paper. We share your commitment and concern to ensuring the resilience of the U.S. financial system.

Best regards,

Chris Kay


Endur comments on "Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U. S. Financial System"

Board of Governors of the Federal Reserve System [Docket No. R-1128]
Office of the Comptroller of the Currency [Docket No. 02-13]
Securities and Exchange Commission [Release No. 34-46432; File No. S7-32-02]

INTRODUCTION

We at Endur have received and reviewed the "Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System" document (and its referenced documents), and believe that this document contains much that will improve the business continuity capability of the financial industry and will help ensure the integrity of our financial markets. The results of the interagency work in this area speak to the importance of the subject and the significant efforts that the agencies and the financial community have expended in this regard to date.

Endur provides critical facility infrastructures that are an essential underpinning of a business continuity plan. We submit these comments and suggestions toward the goal of improving the resilience of our nation's financial marketplaces, and hope you will find them useful.

The Interagency White Paper document (under the section Request for Comments - Sound Practices) asks for comments on several questions including "Should the agencies specify other requirements?" and "Are there alternative arrangements that would provide sufficient resilience in a wide-scale, regional disruption?" We believe we have some valuable answers to both of those questions. Our suggestions relate to: (1) possible requirements of the most critical elements that make up overall reliability and resilience of a location; and, (2) consideration of the formation of co-located community campuses for critical financial marketplaces.

The comments contained herein pertain to market utilities (core clearing and settlement organizations), critical market participants (firms that play significant roles in critical financial markets) and to all six identified critical markets. Both the market utilities and the critical market participants have operations centers, data centers, offices, and in some cases capital market or trading facilities to be deployed and operated. They share much in common relative to their critical facility infrastructure needs for business continuity. They also each have some individual characteristics that need to be addressed separately. We further recognize that the costs involved are significant, and that solutions need to be cost effective.

Our comments are fairly general in their nature since much detail is still unavailable. Specifics are still being determined in many areas, such as:

Endur's comments can be further developed as these items are further defined and we would welcome the opportunity to participate.

SPECIFIC COMMENTS

Our specific comments fall into two categories that address the two questions noted above that were asked in the Interagency White Paper:

  1. Any individual site environment that houses critical facilities must be as resilient as possible - more resilient than most present state-of-the-art data and operations center offerings. Achieving this requires specification of additional requirements for individual critical elements of the infrastructure. Energy, communications infrastructure, security and physical hardening are of particular concern.

  2. The co-location of market utilities and critical market participants (at multiple, redundant sites) in shared financial community campus sites virtually eliminates `back-end' transaction server wide area network (WAN) communications risks, and provides many other possible collaborative business community advantages.

These are both explored in the following paragraphs.

Individual Site Infrastructure Resilience Guidelines

To state the obvious, the more resilient the site, the more reliable, available, and survivable it is. Resilience is the most important aspect of business continuity and is multifaceted. There are many elements that make up the overall resilience of a site including: physical and IT security, vulnerability to natural and man-made disasters, energy and communications network availability, etc. Endur has spent a significant amount of time and money studying the issue of what constitutes true resilience in critical facility infrastructures and how to attain the highest degree of resilience. The following is a brief discussion of each of the critical factors related to site resilience that might profitably be considered by the agencies for specification of additional requirements.

Site

The document asks if distances of 200 to 300 miles between primary and back-up sites might be appropriate. Presumably, these suggested distances are based on general estimates of the disruptive impact of man-made or natural events on the regional infrastructure - the power grid and communications networks, labor pools, local transportation, etc. While prescribing distance is important, there are many other location elements to be considered.

Endur has developed methods for the assessment of critical facility sites and can discuss them with the agencies in more detail upon request.

Structures

The very construction of a critical facility bears on its potential vulnerability. Some potential elements to be considered for specification of guidelines are:

Endur has developed detailed assessment methods for critical facility environment structures, and can discuss them in more detail with the agencies upon request.

Energy

There are many factors involved in the resilience of energy utility services that could be considered as candidates for specification of additional guidelines:

Figure 2. A data center environment designed with integral power and cooling energy generation.

Endur has special expertise in developing assessment methods for energy utility quality and resiliency, and can discuss them with the agencies in more detail upon request.

Security

Physical, information, and intellectual property assets must be secured, as well as the personal safety of employees insured.

Endur has developed methods to assess the security of critical facilities and sites, and can discuss them with the agencies in more detail at their request.

Communications Infrastructure

The resilience and continued operation of the human business processes and the systems operations performed at these sites have critical dependencies upon communications network facilities and services at both a campus and wide area network level.

Endur has developed methods to assess the communications infrastructure of critical facilities and sites, and can discuss them with the agencies in more detail at their request.

Emergency Support Services for the Workforce

Most of the elements we have discussed thus far, other than personal security, have been aimed at the resiliency of the infrastructure that support IT and communications systems operations under emergency conditions. We also need to consider the critical infrastructure that supports the human resources that perform the business processes. People need to eat, drink, sleep, and stay well, etc. to perform their duties, under emergency conditions if necessary.

Endur has developed methods to assess the workforce emergency support services provided by critical facilities, and can discuss them with the agencies in more detail at their request.

Co-location of Collaborating Organizations at Multiple Sites

Resiliency

By co-locating both the market utilities and critical market participants on a multi-tenant campus, resiliency, in terms of the ability to ensure that all transactions that have been initiated are properly cleared and settled within 4 hours, is significantly improved. This is due to the fact that communications between the entities involved would be via a highly reliable Campus Area Network (CAN) instead of the vulnerable, geographically distributed Wide Area Network WAN.

Figure 4. Co-location model for taking business continuity to the next level for financial services.

Consider the possibility of a critical financial processing community on a campus dedicated and specialized to support critical environments for a financial market. All organizations classified as critical to that market would be tenants on this community campus - an exchange trading floor with its data processing and operations facilities, its largest member firms, its clearing and settlement data processing and operations facilities all located together at a hardened campus site. This arrangement would remove the need for WAN connection among and between these entities - and replaced by the local CAN. This could then be expanded to cover multiple financial processing communities since there is significant overlap between communities.

In the co-location scenario, we are changing the patterns of concentration, and the number and location of the resilient sites. Instead of selecting the number and location of sites according to each organization's needs, we select the number and location of resilient sites according to each critical financial community's needs. While this ends up in a smaller (and more cost effective) number of sites, the number and geographic diversity are sufficient to ensure resiliency at an even higher level than in the non co-located scenario.

Co-location of Collaborating Organizations at Multiple Sites

The Interagency white paper expresses the critical need for the completion of clearing and settlement activities for the day and we have provided what we hope will be useful suggestions in that regard. We would be remiss if we didn't further point out that the co-location of organizations at multiple, redundant sites has additional advantages relative to business continuity. By building and operating critical financial community campuses on two, three or more site locations that are widely dispersed geographically and connecting them with secure, highly available proprietary communication links, a far more resilient architecture is created.

This co-location strategy would provide an even higher probability of the timely completion of settlement and clearance, and also continued normal trading activity in all unaffected geographic regions, should the controlling market authorities feel that was desirable. Cessation or restriction of activity in any critical financial market outside of the disrupted region would be a matter of choice, not a technological necessity.

Assuming the number of such financial market community sites was two, one site would still be operational after any wide-scale, regional disruption. The site in the disrupted region, if properly resilient, may well continue to operate, but might lose WAN connectivity, and thus would effectively cease operations after clearing or rolling back all transactions in process. The other site would not be disrupted at all, and organizations' systems at that site could continue to trade over the wide area network to all other regions of the country that were not disrupted. If one expands the example to include all critically classified firms performing financial transactions, then all "critical markets" would be able to continue to trade as well as clear and settle.

Other positive factors for co-location of collaborating parties into campus communities exist. Endur's studies of vulnerability indicate that the safest situation for a critical facility (data center, capital market and trading facility or operations center) is in a properly designed, located and constructed campus environment. Few proper, high quality locations exist - not enough to satisfy the needs of all organizations with critical facility requirements individually. There are thus many arguments for the sharing of such environments amongst multiple organizations on a multi-tenant campus with shared infrastructure components.

In this multi-tenant environment it is cost effective to build the types of resilience that might not be cost effective in a single facility location. While intuition may indicate an increased risk due to the concentration of organizations and infrastructure at one location, the sheer resilience possible at that location and its infrastructure, when combined with a multi-site deployment strategy (each of which collocates each organization in question), actually significantly reduces the risks - and costs! - to every organization individually. Every organization is provided with a network of geographically distributed, hardened and resilient sites.

Many other important potential advantages that could accrue to a Campus community model such as: the co-location of trusted service providers on site for rapid and effective response, or the provision of high levels of personal services and workforce amenities on campus, etc.

Co-location Comments Summary

The redundant, critical financial market community campus architecture described in the co-location scenario, when combined with the resilience intrinsic to each campus when properly specified and designed for energy, communications, and security resiliency, would provide the financial industry with a substantially better approach to business continuity. The approaches presented can also actually be implemented to be more cost-effective despite higher performance and greater features. It would take some leadership within the industry to attain consensus on this approach. Endur has already presented its concept to the critical financial market organizations via direct conversations, proposals, and industry presentations.

SUMMARY

The Interagency White Paper suggests significant improvements in the business continuity requirements for the most significant entities in the clearing and settlement of vital securities. We agree with what has been suggested, but believe there are two additional significant areas that could further insure the business continuity of the securities industry.

  1. A more detailed specification of the desired resiliency requirements for the most critical components of the underlying infrastructure is an important factor in obtaining the level of business continuity desired. This is particularly true in regard to a few of the weakest elements in the present infrastructure - achieving energy, communications and security resilience. We have presented a few ideas in regard to specifying how to achieve that resiliency - achieving independence from the power grid for energy resiliency; adopting Department of Defense-like ground-up design and operations standards for resiliency in security; and the use of both redundancy and diversification of both physical media and communications service providers for resiliency in communications.

  2. The concept of co-location of the involved critical market organizations (at multiple sites) would also be a major enhancement to the industries survivability, particularly in terms of a weak link - wide-area communications vulnerability. These suggestions have many benefits and can be accomplished in a cost effective way.

In both of these areas we speak from experience since Endur's raison d'etre is to create these types of environments. We understand the criticality of your areas of concern and are available to discuss these or other related matters.