This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.

Contingency Plans for PABX and Datacenter

Audit Report No. 226
July 31, 1995

SUMMARY

We are making a number of recommendations to enhance Commission disaster recovery capabilities. The most significant of these are: development and testing of complete and up-to-date disaster recovery plans for the mainframe, networks, PABX, and EDGAR (Recommendations A, C, E, and H); installation of Stratus computers at headquarters to provide a disaster recovery capability for EDGAR (Recommendation D); obtaining back-up power for the headquarters file servers and PABX (Recommendation G); and development of a business recovery plan for the Commission (Recommendation M).

The Offices of Information Technology, Administrative and Personnel Management, and the Executive Director provided comments on a draft of this report (attached). Generally, they concurred with our recommendations.

OBJECTIVES, SCOPE, AND METHODOLOGY

Our objective was to evaluate the adequacy of the Commission's disaster recovery capabilities. The audit scope included mainframe computers and environmental controls at the Operations Center and headquarters data centers; the Private Automated Branch Exchange (PABX) and Electronic Data Gathering and Retrieval (EDGAR) systems; and file servers at the headquarters data center.

During the audit, we interviewed staff from the Offices of Information Technology (OIT); Administrative and Personnel Management (OAPM); and the Executive Director. We reviewed relevant documentation and observed selected environmental controls.

The audit was performed between February and May 1995 in accordance with generally accepted government auditing standards.

BACKGROUND

The Commission's Operations Center in Alexandria, Virginia, has primary responsibility for mainframe and EDGAR operations. The data center at headquarters provides a limited mainframe back-up capability, and also houses network file servers and the Private Automated Branch Exchange (PABX) telephone system. The data centers are linked to each other and regional offices through communication lines.

The OIG observed testing of the headquarters mainframe in a prior audit (No. 208).

The Office of Information Technology has primary responsibility for Commission computer operations. The Office of Administrative and Personnel Management has responsibility for the PABX.

AUDIT RESULTS

MAINFRAME AND NETWORK DISASTER RECOVERY

Disaster Recovery Plans

The Office of User Support Services and the Office of Operations within OIT have developed separate, but overlapping, disaster recovery plans. User Support Services' plan is not stored off-site, reducing its accessibility. The plans are not complete and up-to date.

Recommendation A

OIT should develop one disaster recovery plan which is complete and up-to-date, after consultation with affected offices. Copies of the plan should be stored off-site.

Risk Assessments

OIT has not performed a risk assessment of the Operations Center. A risk assessment is needed to identify vulnerabilities and evaluate appropriate safeguards.

Recommendation B

OIT should perform a risk assessment of the Operations Center.

Testing

The two plans have not been periodically tested, because of other priorities. User Support Services' plan was last tested in November 1994, while Operations' was tested in June 1994. Since June, Operations has implemented a major system upgrade.

Recommendation C

OIT should test and update its disaster recovery plan(s) periodically. Other offices should participate in the tests, as appropriate.

EDGAR

Disaster recovery

OIT recently acquired additional Stratus mini-computers. After testing them, it plans to install them in headquarters, thereby providing a disaster recovery capability for EDGAR. OIT has not yet developed an EDGAR disaster recovery plan.

Recommendation D

OIT should test and install the Stratus mini-computers, as planned.

Recommendation E

OIT should develop a disaster recovery plan for EDGAR, after consultation with affected offices.

Re-solicitation

The EDGAR contract will be re-solicited in 1997. The new contract should provide for a continued disaster recovery capability.

Recommendation F

OIT should require the next EDGAR contractor to provide a disaster recovery capability.

HEADQUARTERS PABX AND FILE SERVERS

Back-up power

Currently, the PABX and file servers at headquarters do not have a back-up power source in the event of a power interruption. A generator is available, but it is not in operating condition. Unlike the mainframe, the PABX and file servers do not have an alternative site.

Recommendation G

OAPM should put the generator in operating condition.

Disaster recovery plan

OAPM has not yet developed a contingency plan for the PABX. The Appendix contains suggestions for the contents of a plan.

Recommendation H

OAPM should develop and test a PABX contingency plan, after consultation with affected offices.

ENVIRONMENTAL CONTROLS

Water

Water has leaked into the Operations Center computer room from a drain pipe on the floor above. An ice machine and sink share the same drain, which periodically clogs. Water can damage computers and cause electrical shocks.

Recommendation I

OAPM should prevent further leaks into the Operations Center computer room, for example, by relocating the sink and ice machine.

Traffic Reports

OAPM has installed a system for monitoring traffic on the PABX. Review of this information can help ensure that the PABX's capacity is sufficient.

Recommendation J

OAPM should review PABX traffic reports.

Maintenance Logs

OAPM does not keep maintenance logs for the PABX and environmental control equipment. Maintenance logs help ensure that required maintenance is performed.

Recommendation K

OAPM should keep maintenance logs for its PABX and environmental control equipment.

Blueprints

In the event of a disaster, OAPM would need to consult blueprints of the headquarters and Operations Center facilities. To ensure that blueprints are readily available, a copy should be stored off-site.

Recommendation L

OAPM should store a copy of the headquarters and Operations Center blueprints off-site.

OTHER MATTERS

The Commission does not have a business recovery plan for its overall operation (as opposed to just its computer operations) in the event of a major disaster. This plan would help ensure that the Commission's programs are promptly and effectively restored after a disaster. The plan would explain what should be done, and who should do it.

Recommendation M

The Office of the Executive Director should develop a business recovery plan for the Commission, as resources permit. It should consult with other Commission offices and divisions.

APPENDIX

The following elements should be included in the PABX contingency plan:

a) OAPM should document in their contingency procedures the location and phone numbers of all analog lines in the Headquarters and Operations Center buildings for possible use in the event of a main PABX failure. Additionally:

b) OAPM should ensure that vendor contact information such as switch manufacturer, contract support providers, the D.C. and Virginia phone companies, and cellular phone providers, and equipment inventory is periodically updated in the plan.

OAPM should also include contact information that can assist the telecommunications staff in the event of a catastrophic disaster (For example, it may include GSA's NSEP Division).

c) OAPM should work with program offices to determine the minimal number of telephone numbers/lines needed at headquarters, Operations Center and Annex to support program office continuity in the event of a PABX communications failure/unavailability and include this in their contingency plan.

d) OAPM should document in their contingency plan the procedures to be used to notify/forward calls to new Commission numbers.

e) Copies of PABX manuals, and identification numbers of Central Office to PABX and inter-switch trunk lines should be maintained at both the Operations Center and at headquarters. The location of manuals and the identification of lines should be in the contingency plan.

• - OAPM should ensure that an adequate number of analog phones can be readily obtained/available to use these lines.

  - Existing users of these analog lines should be informed that these lines may be taken away in the event of an emergency to restore Commission telephone support capabilities.

  - Arrangements should be made among program offices for the use of analog lines located in other program offices.