Level 3 Communications, Inc.
Jennifer J. Johnson
Re: Docket No. R-1128
Office of the Comptroller of the Currency
Attention: Docket No. 02-13
Jonathan G. Katz
Re: File No. S7-32-02
Level 3 Communications, Inc. respectfully offers comments to the Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System. Level 3 understands that the strengthening of the U.S. Financial System will rely heavily on financial corporations, their procedures and highly trained personnel in multiple locations. Our core competencies do not lie in financial activities, however we believe we can offer a highly qualified view of the physical and data network infrastructure, necessary to the core clearing and settlement organizations to perform critical activities even in the event of wide scale disruptions.
Network providers should routinely perform a comprehensive review of all potential security threats that could jeopardize the operation of services. Security and Business Continuity Plans need to address all categories of threats including:
Level 3's comments will specifically address network and data elements that are critical to operations and should be scrutinized when evaluating the resiliency and security of network infrastructure providers. Key topics will include:
Level 3 Communications (Nasdaq:LVLT) is an international communications and information services company. The company offers a wide range of communications services over its U.S, trans-Atlantic and European 20,000 mile broadband fiber optic network including Internet Protocol (IP) services, broadband transport, colocation services, and patented Softswitch-based managed modem and voice services. Its Web address is www.Level3.com.
If a provider has a network without complete route diversity, they will not have the capability to provide route-protected services. In the case of a common fiber cut, mission-critical services are vulnerable to outages.
It is extremely important to carefully review a network provider's physical topology maps. Demand to see detailed maps from providers that show how they enter and leave a given market, including specific buildings within that market. Be certain that their topology is a true representation of their backhaul and metropolitan networks. "Logical" topology maps are not always a true physical representation of a network and the routes that comprise the physical infrastructure.
Metro Route Diversity
Fiber cuts are more likely in urban areas. If a network provider's points of presence do not have diverse and redundant connectivity to their backbone network, they are exposed to outages. This exposure could be significant, because data centers serve as aggregation points for local/metro connectivity where colocation services are usually provided. Without diverse routes from these points, service outages are more likely.
Complete network ownership translates into effective control over potential problems. A uniform platform, constructed with consistent technology, limits the variables technicians face when troubleshooting network components. This directly translates into operational efficiency and minimizes the potential for component failure.
Terrestrial fiber is much more susceptible to natural and man-made disasters when exposed on poles or buried only slightly beneath the surface. Fiber that is shielded and buried several feet beneath the surface across an entire network would provide the highest level of security and protection.
An all fiber-optic network offers advantages over older technologies for protecting data transmissions. Fiber-optic cables are difficult to tap, and it is more difficult to access data transmitted through fiber than via signals over copper wires.
Backup power is critical to keep the network operating in the wake of power failures or outages. Battery backup can handle short interruptions, but longer outages require backup generators. Without these backup systems in place, the network availability will be severely degraded.
Each Data Center should have an uninterruptible power supply (UPS) provisioned in an N+1 configuration. Battery back-up and emergency generators should provide enough backup power to keep facilities operational at maximum load. The UPS batteries should support all customer equipment and all essential equipment of the facility.
In order to ensure network diversity through multiple carriers, a network provider should create a business environment that allows unencumbered access to any desired carriers' services.
As the most visible form of a network, data center sites must be designed to withstand man-made and natural disasters, as well as provide security for both critical equipment and employees. Stringent physical security policies and controls should be employed as to not allow unescorted access to the colocation areas. All perimeter doors should be alarm-monitored with video surveillance. Photo ID Access Cards and an access control system that supports 24/7 monitoring and logging of all entryways should be provided. Access records should be stored in the event the records are required for non-repudiation. In addition, biometric scanning devices are recommended.
At each data center location within the carrier's network there should be at minimum dual fiber entrances. These fiber entrances should always be on separate sides of buildings where fiber paths are diverse. Without dual entrances there are increased risks to lose connectivity to an entire data center from the network.
One of the biggest problems facing the telecommunications industry today is an inability for providers to have an accurate view of their available inventory of capacity. Often, orders will be accepted and promises made for delivery without any real verification that the capacity requested is available in the network. The results of this are the late delivery of services by the provider. With a "real time" view into networks, the user can operate their infrastructure with greater flexibility for operations, capacity planning, and service management. In the event of a wide scale disruption it will be essential to have the ability to redesign and provision circuits in real time.
Some networks that have been created by acquiring capacity from multiple sources are more susceptible to human error. They're built with different varieties of equipment, and multiple management systems increase the complexity of operations and require a higher level of training for the staff. Networks monitored by multiple carriers can delay fault isolation because there is not a cohesive team or network to monitor and troubleshoot for fault isolation. Coordination of network testing becomes more complicated when multiple network monitoring staffs need to work together for these functions.
The factors that drive network availability range from physical layer diversity in the network, an ability to fix fiber cuts quickly, and an operations system that allows quick fault isolation on the network in order to accelerate repair time. In reality, many providers rely on non-diverse physical layers and outsourced field technicians. All these factors will drive down network reliability and associated availability.
After acquiring a provider's services consider the level of engineering support when a problem arises. Without immediate interaction with an engineer who is trained to at least Level 2 technical support, the risk of misdiagnosis or excessively slow repair times increases significantly.
Trusted employees and vendors are essential to maintaining the highest network operational reliability and availability levels. Prior to selecting a network provider the purchaser should demand that the provider's employees and vendors meet stringent background screening standards for trustworthiness, honesty and reliability. Policies and standards should be designed to ensure that only employees and vendors who meet the highest personnel security standards are authorized access to the network and critical support facilities.
If a provider is supplying services with a meshed protection scheme, it is imperative that the entire network is designed with a true meshed capability. The most important point here is that mesh architectures save cost by using shared protection capacity. Consequently, the provider might engineer the network for complete restoration in the event of a single failure or fiber cut. But what happens when there are two failures? How is the protection capacity allocated?
Protected private line services are often the platform for mission-critical activities. It is extremely important to be sure that a private line service has its protected path built with a completely diverse route. If the protected path is built on the same path on which the primary circuit is delivered, then the service will suffer an outage in the event of a fiber cut. Providers will often oversubscribe SONET rings, which means protection paths can be eliminated to provide primary paths.
Network providers need to ensure sufficient capacity for normal and fail-over operations by limiting every internal network link to less than 50 percent utilization. Limiting link utilization to less than 50 percent guarantees redundant capacity in the event of failure of other network links.
To ensure multiple levels of redundancy, wide area packet networks should be designed so every Label-Switched Path (LSP) is pre-provisioned with primary, secondary, and tertiary paths. If the primary LSP fails, traffic would be automatically rerouted to the secondary LSP, typically within two seconds. If the secondary LSP also fails and a tertiary LSP exists, traffic is automatically rerouted to the tertiary LSP. If all LSPs fail, traffic routing falls back to normal, redundant IP routing, if it is running on a MPLS (Multi-Protocol Label Switching) core.
Layer 3 VPN does not differ from a Layer 2 VPN from a technical perspective. Once a VPN has been established it follow very similar paths at the physical layer. The main differences are at the creation of the DMZ (De-Militarized Zone).
Layer 3 (RFC 2547) VPNs are very CPU-intensive, running many VRFs will result in higher CapEx cost in building infrastructure. Layer 3 VPNs are also difficult to manage, with multiple boxes responsible for different segments of the DMZ again resulting in higher OpEx cost.
Protecting the U.S. Financial System from any disruption is a critical challenge both financially and operationally. Level 3's comments to the Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System are designed to offer insights into the risk mitigation that network providers should employ in their physical infrastructure and network designs to protect the U.S. Financial System. Cost effective implementation of programs that ensure the survivability of critical activities and functions of core financial companies will be an ongoing concern. A better understanding of the potential and avoidable risks inherent in network providers should off set some cost incurred today and in the future.
100 William Street
New York, NY 10038
Office (212) 487-0109
Cell Phone (914) 645-5964
Pager (888) 561-7991