Wachovia CorporationOctober 21, 2002 Ms. Jennifer Johnson, Secretary
Ms. Christine Tomczak
Office of the Comptroller of the Currency
Johnathan G. Katz, Secretary
RE: Draft Interagency White Paper on Sound Practices to
Ladies and Gentlemen:
Wachovia Corporation1 and it subsidiaries (collectively "Wachovia") welcomes the opportunity to comment on the "Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System" ("White Paper") to the Board of Governors of the Federal Reserve System, the Office of the Comptroller of the Currency, the Securities and Exchange Commission, and the New York State Banking Department (collectively the "Agencies"). Wachovia applauds the Agencies' efforts to identify potential causes for wide-scale regional disruptions to critical domestic wholesale financial markets and, through regulatory oversight and guidance, prevent such disruptions from adversely affecting critical wholesale financial markets. Wachovia supports many of the themes and key concepts outlined in the proposal, but wishes to provide the following comments for your consideration.
Responses to the Request for Comments
1. Definitions.
Wachovia requests that the Agencies both adopt clear definitions and articulate helpful examples of the activities to be covered under such definitions. For example, although we have a general understanding of what "core clearing and settlement organizations" means, we are unclear whether or not, and the extent to which, federal systems such as Federal Wires would fall within that definition and thus be held to the same standard. Similarly, we request more explicit guidance concerning the organizations that would be considered to "play significant roles in the critical financial markets." Absent a clear description of which organizations would be deemed "significant," it would be difficult to determine whether or not critical markets or organizations might inadvertently be excluded. Clearer guidance on "other critical infrastructure components" and "other geographic areas" is necessary to help affected organizations assess the impact of any final regulatory guidance on their business resumption practices.
The Agencies also request comment on the definition of a "wide-scale regional disruption." In our view, the current proposal does not provide a sufficient description of the events that would fall within this definition. A definition based solely on time duration would not be appropriate because, at the time an incident occurs, it would be unclear how long the disruption might continue. Similarly, it is unclear what factors might cause an incident to be deemed "wide-scale." It should also be noted that any definition based on an incident's initial impact or scope of disruption may not consider other factors that might trigger an organization's business continuity plan. For these reasons, Wachovia recommends that the Agencies consider the use of variable factors, such as type, extent and duration, to determine which events should constitute a "wide-scale regional disruption."
Wachovia recommends that the White Paper also apply to (i) any organization that processes settlement transactions (e.g., the New York Stock Exchange; Sunguard; etc.) and (ii) any market participant that processes settlement transactions on behalf of other financial institutions. This would ensure that market participants would be prepared for wide-scale regional disasters and, therefore, promote stability in the wholesale financial markets. Processing organizations not covered by the White Paper may fail to participate in settlement processing. Such an event would likely cause significant financial instability within an affected region because of the increased dependency on the remaining organizations participating in settlement processing.
2. Recovery and Resumption of Critical Activities.
The Agencies request comment concerning emerging industry objectives that (i) "core clearing and settlement" organizations establish a "resumption-time target" of no later than two hours after a disruption; (ii) companies with core clearing and settlement responsibilities not only recover but also fully resume operations during the business day when a disruption occurs; and (iii) that other organizations deemed to have a "significant role" in the wholesale market establish a resumption-time target of no later than four hours after a disruption.
Wachovia strongly supports aggressive resumption efforts. Concerning the proposed timeframes, however, it is critical to recognize that an organization's ability to resume activities within several hours or a business day may depend on a variety of factors. These factors may include the distance between data centers; the location of the command center to the data centers; staffing; telecommunications, transportation and other dependencies; and the size and duration of the disruption. We believe that effective resumption timeframes must reflect these and related factors.
An effective recovery time standard for an organization must be sufficiently flexible and incorporate a number of factors including the size of the organization, its customers' expectations, the markets supported, and an organization's dependencies. "One-size-fits-all" recovery timeframes do not reflect the actual market constraints affecting an organization or the volumes of transactions that must be processed. In other words, what might be an achievable recovery timeframe under one set of factors or in one environment may not be transferable to, or achievable under, another set of factors or in another environment.
Furthermore, any intraday recovery requirements that are established will result in significantly increased demand on the interfacing applications that support settlement processing. Although the Agencies did not request comments concerning these various dependencies, Wachovia believes these factors must be considered in order to establish effective recovery timeframes.
3. Sound Practices - Data Center Locations.
The Agencies seek comment concerning out-of-region back up facilities and, in particular, whether or not there should be a minimum distance between the primary and back up locations. Wachovia believes that distance alone should not be a determining factor, but that it should be one of a number of factors that an organization might consider to determine primary and secondary locations for data centers.
Certain locations, regardless of distance between them, may not be good choices for redundant data centers because of the potentially high risks from from either man-made threats, natural threats or both. Placing primary or secondary data centers in close proximity to high-risk areas increases commensurately the risks associated with a data center's location. These factors, in our view, are far more substantial than any distance requirement.
Wachovia believes that primary and secondary data centers in high-risk areas should be located in separate geographic "infrastructure zones," each defined by a zone's infrastructure capabilities (e.g., electrical grid, telecommunications, water, transportation, etc.). Thus, should any data center be located in a high-risk area, none would be dependent on identical infrastructure elements. Note that this location strategy is not determined by distance between sites, but rather by each site's ability to support the needs of the organization during a disruption.
Moreover, the need for high availability is an important component of a sound business plan. One means to increase availability is to place primary and backup data centers within a reasonable distance of one another. Significant distances between primary and backup data centers may reduce or eliminate an organization's ability to mirror data synchronously in the high volume, performance-based transaction processing environment required by market makers and major institutions.
Given current technical constraints, distance requirements may result in data being mirrored asynchronously. In a primary site failure, data loss may occur for in-flight transactions and thereby prevent intra-day recovery. The use of tape or cartridge backups may be available to recover in-flight transactions when data centers are geographically dispersed. This recovery method is often tedious, time consuming and typically is used only in severe circumstances. Another option employing existing technology would be to process simultaneously, by executing each transaction twice, those applications with a high recovery point objective through primary and secondary centers. In addition to increasing data integrity risk, simultaneous processing would likely pose significant additional financial burdens on an organization and its customers. In addition, data centers located in widely separated geographic locations may increase the time necessary for an organization to recover to an acceptable level.
For these reasons, Wachovia believes that any final guidance permit each organization to consider its technology limitations and other unique factors when developing its data center strategies.
4. Sound Practices - Labor Pools.
The Agencies request comment concerning dispersed labor pools. Wachovia encourages the Agencies to evaluate the benefits of segregating data centers separately from command centers. Employees necessary to support business continuity programs may be located in facilities separte from the data centers. This is accomplished by operating data centers in a "lights-out" or "lights-dim" environment, which requires either no, or very limited, operational staff to be located at a data center. By separating data centers and command centers, organizations can minimize the adverse impact of a disruption.
Wachovia believes that organizations should be able to use a risk-based approach to determine the necessity for maintaining duplicate labor pools to support application and processing activities associated with critical wholesale markets. Frequently, individuals supporting these functions must have highly-specialized technical expertise. To avoid redundant staffing support in multiple locations, the White Paper should provide an organization broad latitude to address the human resource aspects of its business continuity program.
5. Timetable for Developing Plans and Implementation.
The Agencies request comment concerning whether or not specific timeframes should be established for implementation of any new standards. Wachovia supports articulation of a reasonable timetable for developing or revising any business continuity plans. However, we find the proposed 180 day period for the adoption of revised implementation plans is far too little time. Wachovia recommends that the Agencies postpone establishing the time period for the adoption of revised implementation plans until after issuance of the White Paper. This would provide affected organizations an opportunity to develop realistic timeframes to revise their business continuity plans. At a minimum, affected organizations should be allowed at least 18 months after issuance of the White Paper to revise their business continuity plans.
Similarly, an organization's ability to meet any mandatory compliance date for implementing its revised business continuity plans will likely depend on the degree to which the White Paper is consistent with the organization's existing business continuity plans. Accordingly, Wachovia supports the Agencies' position that organizations incorporate any new standards set forth in the White Paper as soon as practicable. This would permit organizations to make necessary changes to their business continuity programs without jeopardizing day-to-day operations. Wachovia does not believe that mandatory compliance dates will increase the speed or effectiveness with which organizations will implement their revised plans.
Additional Comments
1. Routinely Use or Test Recovery and Resumption Arrangements.
Wachovia agrees with the Agencies' position that testing is an important part of any business continuity program. We recommend, however, that systems that have successfully demonstrated failover capacity be excluded from additional testing requirements unless additional dependencies exist that were not adequately tested as part of their regular failover activities.
2. Flexibility of Solutions.
Wachovia also supports the Agencies' view that there is no "one size fits all" solution to business continuity planning and implementation, and that organizations should achieve sound business continuity practices as soon as practicable. The ability, functionality and location of the primary and secondary data centers, as well as the ability to hire and train essential staff, are important factors that must be considered to develop effective business continuity programs. Wachovia believes that the need to recover from the majority of outages must be balanced against the need to recover from the most severe disruptions. Business continuity programs should support both of these goals. Future enhancements to business continuity programs should not come at the expense of these goals.
3. Costs.
Should the Agencies mandate specific distance requirements, the financial burden will likely fall on "core clearing and settlement" organizations or those that have "significant roles" in wholesale financial markets. For example, the cost of a new, state-of-the-art data center is estimated at approximately $200 million plus operating costs including personnel and telecommunications. Accordingly, organizations required to comply with any new data center distance requirements will likely incur significant additional costs.
4. Service Providers.
Organizations frequently rely upon third-party vendors to support applications for business continuity plans covered under the White Paper. Many of these service providers are not regulated by any of the Agencies. These dependencies will affect both recovery timeframes and sound business continuity planning. Wachovia recommends that the Agencies consider these factors in formulating any new requirements.
5. Coordination between Agencies.
Wachovia also recommends that the Agencies coordinate their business continuity efforts with other regulatory agencies, including the NYSE and NASD, as well as state and local agencies that may have separate business continuity standards for organizations identified in the White Paper as having "core clearing and settlement" responsibilities or "significant roles" in the wholesale financial markets.
Very truly yours,
/s/
Michael A. Watkins
cc: Jean Davis
____________________________
|