This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.

Year 2000 Compliance Efforts

Audit No. 285

August 24, 1998

EXECUTIVE SUMMARY

The Office of Inspector General is continuing to audit the Commission's efforts in making EDGAR and its internal computer systems year 2000 compliant. This report covers the period from May 1996 to June 15, 1998.

For this period, we found that the Commission has taken numerous steps to address the year 2000 issue. It has developed plans, instituted a year 2000 organizational structure, obtained senior management involvement, identified contract vehicles for year 2000 work, prepared status reports, and begun remediation¹ of systems, among numerous other steps.

Like many other federal agencies, the Commission is having difficulties meeting year 2000 deadlines established by the Office of Management and Budget (OMB). One major reason is long-standing weaknesses in the Office of Information Technology (OIT), which is primarily responsible for year 2000 remediation on mainframe systems. OIT is undergoing a major reorganization to address those problems.

We are making several recommendations to improve the Commission's compliance efforts. These include hiring a consultant, developing an overall management plan, clarifying the organizational structure, prioritizing critical systems, adding additional skills to the project team, developing a contingency plan for EDGAR testing, and providing additional technical assistance to offices and divisions. OIT has already identified many of these needs, and is currently addressing them, as explained below.

SCOPE AND OBJECTIVES

Our objective is to evaluate the progress of the Commission in making EDGAR and its internal systems year 2000 compliant. Since the Commission's efforts are ongoing, we intend to issue several reports, of which this is the first.

The period covered by this report is from May 1996 (when the Commission's year 2000 efforts began) to June 15, 1998. The report consolidates work from three year 2000 audits: EDGAR (Audit No. 275), OIT internal systems (Audit No. 274), and non-OIT internal systems (Audit No. 282).

We used several sources as criteria for our review, including "Year 2000 Readiness Kit" by the Information Systems Audit and Control Association, "Year 2000 Computing Crisis: An Assessment Guide" by the General Accounting Office (GAO), "Year 2000 Computing Crisis: Business Continuity and Contingency Planning" (draft) by GAO, "Year 2000 Certification Program Questionnaire" by the Information Technology Association of American and the Software Productivity Consortium, and Year 2000 "Best Practices" on the General Services Administration's (GSA) web site.

During the audit we interviewed staff, reviewed project plans, status reports, and other relevant documentation, and conducted a survey of offices and divisions remediating PC systems. We did not evaluate the accuracy or adequacy of Commission reports provided to the Office of Management and Budget (OMB) and the Congress. GAO recently issued an audit report that included two recommendations for improving Commission reports to the Congress.²

The audit work was performed from November 1997 to June 1998 in accordance with generally accepted government auditing standards.

AUDIT RESULTS

Our audit findings and recommendations are presented below. First are general issues that affect all of the Commission's remediation efforts. Then we present specific findings from each of the three audits we are performing: EDGAR, OIT internal systems, and non-OIT internal systems.

GENERAL ISSUES

STATUS OF COMMISSION EFFORTS

The Commission has made significant efforts to address the year 2000 problem and plans to increase those efforts during the next year and a half. It has developed an organizational structure and some plans for the year 2000 project, developed an inventory of internal systems to be made compliant, obtained senior management involvement, identified contract vehicles for testing and renovation of systems, prepared status reports,³ and begun remediation efforts, among other steps.

Like many other federal agencies, the Commission has had difficulties meeting OMB deadlines for completing various actions in the remediation process. One major cause of the delays is long-standing weaknesses in information resources management (IRM). These weaknesses include project management, policies and procedures, planning, security, and staff skills. OIT is implementing a major reorganization and expanded use of out-sourcing to address these issues.

Given the IRM weaknesses and the limited time available, hiring a consultant well qualified in year 2000 testing and remediation is advisable. Among other possibilities, the consultant could provide options to expedite remediation, help improve project management techniques, and assist in drafting procedures and technical materials.

According to OIT, the CIO and other OIT staff have met several times with the Chairman, the Chairman's Chief of Staff, a Commissioner, and their key staff. At these meetings, OIT has provided status reports on the Commission's year 2000 project, as well as plans for contractor assistance.

Recommendation A

OIT should consider hiring a qualified year 2000 consultant.

OIT response: OIT indicated to us that it is currently in final negotiations with a consultant for program management assistance and independent verification and validation (IV&V) services.

MANAGEMENT PLAN

The Commission has developed plans for some specific year 2000 activities (e.g., EDGAR compliance) and is developing an overall year 2000 management plan. As indicated in GSA's best practices web site, such a plan should include a brief statement of the problem, objectives, management strategy (e.g., centralized management with decentralized execution), and assignment of responsibilities. A high-level management plan would assist in developing more detailed year 2000 project management plans.

Recommendation B

OIT should finalize its overall year 2000 management plan. It should provide the plan to the Executive Director, the Chairman, the EDGAR Steering Committee, and the Information Technology Capital Planning Committee for their review and approval. The plan, when approved, should be made available on the Commission's Intranet.

OIT response: OIT indicated that it has drafted a plan and expects to finalize it soon. IT stated that copies of the draft have been provided to the Chairman's Office, another Commissioner, the Office of Inspector General, and the Office of the Executive Director, as well as several year 2000 support vendors. One of these vendors called it "highly workable."

ORGANIZATIONAL STRUCTURE

According to the Commission's May 1998 status report, the responsibility for ensuring that Commission systems are year 2000 compliant rests with the Executive Director and the Associate Executive Director for Information Technology (the Chief Information Officer or CIO, who also heads OIT). The Executive Director reports directly to the Chairman; the Associate Executive Director reports to the Executive Director.

A task force within OIT has day-to-day responsibility for the year 2000 project. In addition, a Year 2000 Project Coordinator has record-keeping and reporting responsibilities. The Project Coordinator reports to the Executive Director.

Another potential participant in this project is the Information Technology Capital Planning Committee. The Committee was established in December 1997 in response to federal legislation seeking to improve management of information technology. The Committee recently held its first meeting. Its members consist of the Executive Director, the Chairman's Chief of Staff, the CIO, and the Chief Economist. Strategic Year 2000 issues would appear to be within the purview of this Committee.

The Chairman has recently designated a member of his staff to monitor the status of the year 2000 project. The Chairman has also asked another Commissioner to take an active role in ensuring that the project is successful.

Because numerous entities have some responsibility for year 2000 issues, the Commission needs to clarify its year 2000 organizational structure to avoid duplication of effort and miscommunication.

Recommendation C

The Office of the Executive Director, in consultation with the Chairman and other affected entities, should decide what organizational structure and assignment of responsibilities for year 2000 issues will be most efficient and effective.

YEAR 2000 PROJECT TEAM

The Year 2000 project team was established in May 1996. At first, the team had eleven members from all parts of OIT, who participated on a collateral duty basis. In January 1998, a team of five full time members was established.

The core team does not have members with certain skills it needs. These skills include ADP procurement, database administration, data communications and networks, systems programming, and technical writing. Also, it does not include user representatives. Given the urgency and importance of year 2000 remediation, the skills and perspectives on the team should be broadened as soon as possible.

Recommendation D

OIT should include on the year 2000 project team members with additional skills and representatives from user offices.

OIT Response: OIT indicated that these skills are not required on a full time basis and are currently provided as needed from the subject matter areas. This has proven to be an effective solution. OIT also indicated that it has pursued a "matrixed" approach to user participation.

STATUS REPORTS

OIT has prepared status reports as required on the year 2000 project for OMB, the Congress, and the Chairman. To ensure that the project proceeds as planned, OIT also needs to develop regular (e.g., monthly) internal reports containing detailed milestones and the status of remediation for all critical systems.

Recommendation E

OIT should develop detailed reporting mechanisms for the status of year 2000 remediation.

OIT Response: OIT indicated that the consultant it plans to hire will help it develop improved reporting techniques for senior management. OIT currently produces numerous spreadsheets and status reports at the system level.

BUDGETING

Normally, costs of year 2000 remediation are estimated in terms of lines of code, with some other factors considered. OIT has not yet decided on a methodology for estimating these costs. For example, it used a contractor bid for remediating nine applications as a basis for estimating costs for remediating eighteen other applications. This methodology assumes that remediation costs are comparable among systems, which may not be true.

Current methods have led to a tendency to under-estimate costs. Commission cost estimates for year 2000 remediation have been continually rising (a general problem in the information technology industry).

Recommendation F

OIT should decide on a methodology for estimating costs for year 2000 remediation and use this methodology in preparing cost estimates. The methodology should be generally accepted within the government or industry, or be an industry best practice.

OIT Response: According to OIT, its IV&V contractor has worked with it to develop standardized metrics for year 2000 reporting and progress measurement.

COMPLIANCE TESTING POLICY

The Commission does not have a final testing (i.e., certification) policy for software and hardware systems being made year 2000 compliant. Without a policy, the extent of testing may be insufficient or excessive.

Generally, testing should increase as risk increases. For example, a mainframe system with multiple users should be tested more rigorously than a non-critical stand-alone PC system. A recently developed non-critical PC application which the vendor certifies as compliant might need only cursory testing.

Some products may not be feasible to test (e.g., operating systems), in which case year 2000 compliance will be based on vendors' warranties. This issue would also be covered in the testing policy.

Recommendation G

OIT should finalize a testing (i.e., certification) policy for systems being made compliant.

CONTRACT LANGUAGE

To ensure compliance with the final Federal Acquisition Regulation rule on the year 2000, the CIO Council Sub-Committee on the year 2000 sponsored development of recommended contract protection and warranty language. The intent of the language, which was issued on August 22, 1997, is to ensure that hardware and software products acquired by the government are year 2000 compliant. OIT and the Office of Administrative and Personnel Management (OAPM) indicated that they are now using the recommended language in contracts.

Recommendation H

OIT and OAPM should ensure that the recommended contract language is included in solicitations and contracts for year 2000 software and hardware.

EDGAR FINDINGS

EDGAR YEAR 2000 COMPLIANCE (AUDIT MEMORANDUM NO. 8)

Our office has already issued an audit memorandum on EDGAR year 2000 compliance (No. 8, attached in the Appendix). In its response to that memorandum, OIT discussed EDGAR test plans. Briefly, it intends to treat EDGAR primarily as a replacement system under development rather than as a legacy system in need of renovation.

OIT's rationale is that most of EDGAR is scheduled to be replaced by January 2000 due to the proposed EDGAR modernization. It stated that the risks associated with EDGAR and year 2000 compliance are considered quite low.⁴

In our audit memorandum, we noted that the Commission's EDGAR test plan is not in conformity with OMB target dates. We recommended that OIT obtain senior management acceptance of the risks posed by non-compliance with the target dates.

We also noted that OIT planned to use contractors during 1998 and 1999 to perform testing of EDGAR. Because contractors may not be available during those time periods, we recommended that OIT lock-in necessary contractor staff through a binding contract.

INDEPENDENT VERIFICATION AND VALIDATION

OIT's "EDGAR and Year 2000 Testing" plan, dated March 6, 1998, indicates that a contractor will perform independent verification and validation (IV&V) of new EDGAR code for the EDGAR modernization. According to the plan, this contractor will review plans, activities, and results pertaining to the development testing performed by the EDGAR contractor.

The scope of review as stated in the test plan does not meet standards for IV&V developed by the National Institute of Standards and Technology (NIST).⁵ These standards require the IV&V team be independent technically, managerially, and financially. As relevant here, managerial independence requires the team to independently decide what software to analyze and test, the techniques used, and the technical issues to act upon (rather than limiting review to tests performing by the development contractor).

The test plan also does not make clear that the IV&V contractor will be present at the beginning of the development process, as required by the standards, rather than being brought in at the end. Software verification and validation is to be tightly integrated with software development under NIST standards. Early inclusion of the IV&V contractor will promote more effective testing.

Recommendation I

OIT should modify the EDGAR test plan to conform to NIST standards for IV&V and to indicate that the IV&V contractor will be present at the beginning of development.

CONTINGENCY PLAN

OIT's response to our audit memorandum (see above) indicated that it would not be cost effective to test existing EDGAR code (old code) that will be replaced by January 1, 2000. Instead, OIT plans to have a contractor do a code review of the existing EDGAR code. It believes that a code review will be considerably cheaper and more appropriate in this case than testing. The modernized EDGAR code (new code) will be thoroughly tested for year 2000 compliance.

OIT's plan appears to assume that the EDGAR modernization will proceed on schedule without delays. A more realistic assumption, given prior OIT experience with EDGAR and other system development projects⁶ is that there will be delays (for example, the EDGAR redevelopment contract was originally scheduled to be awarded by January 1997, but is now scheduled for award in June 1998). However, the current schedule does not have any room at all for delays, since January 1, 2000 is not a deadline that can be moved.

Therefore, OIT needs to have a contingency plan in case of delays. The plan would provide for full scale testing of non-replaced old code when OIT determines that portions of EDGAR will not be modernized by the millennium date. It should define when OIT would make that determination (early enough to readily arrange for the tests).

Recommendation J

OIT should develop a contingency plan for testing (i.e., certification) of existing EDGAR code if the modernization of EDGAR experiences delays.

OIT INTERNAL SYSTEMS

MISSION CRITICAL SYSTEMS

The Commission has designated 90 enterprise (generally mainframe) systems to be made compliant, of which 53 are mission critical and 37 are non-mission critical. These designations were based on responses from users, rather than from an independent, formal process.

Commission officials have acknowledged that the number of systems deemed to be mission critical is excessive, especially considering the relatively little time available for remediation. They plan to identify the 8-10 most critical systems.

Focusing on the most mission critical systems should help OIT prioritize its workload. However, the systems no longer deemed mission critical should still be considered a higher priority than the 37 systems originally designated non-mission critical.

Recommendation K

OIT in consultation with the Chairman's Office, the Office of the Executive Director, and other affected entities, should consider establishing several categories for year 2000 systems to help prioritize its workload (e.g., mission critical, significant, and not significant). The systems formerly designated mission critical would be redesignated as significant.

AUTOMATED INVENTORY TOOLS

OIT is spending significant resources to develop an accurate inventory of hardware and software. Several automated tools are available that could assist in this task.

Recommendation L

OIT should use an automated inventory software tool.

OIT response: OIT indicated that it has already made plans to employ a suitable validation and data collection product.

NON-OIT INTERNAL SYSTEMS

MISSION CRITICAL SYSTEMS

According to the Commission's May 1998 status report, 88 PC-based applications are being remediated. Generally, the divisions and offices that developed these systems (not OIT) are responsible for their remediation. OIT has provided, and will continue to provide, technical assistance, but its resources are already stretched.

Of the 88 systems, 53 have been designated as mission critical and 35 as non-mission critical. The designations were made based on user responses, rather than an independent, formal process.

The number of mission critical systems may be excessive, especially considering that these PC applications are generally used by only one organization (also see our similar finding above under OIT internal systems). Prioritizing the number of mission critical systems would allow scarce resources to be concentrated on the most important systems.

Recommendation M

OIT, in consultation with the Chairman's Office, the Office of the Executive Director, and other affected entities, should develop criteria for assessing the significance of PC-based applications. Based on this criteria, OIT and user offices should reassess which PC based applications are mission critical. OIT should also consider establishing several categories for PC-based applications (e.g., mission critical, significant, and not significant).

TECHNICAL EXPERTISE

We surveyed ADP liaisons who will be performing remediation of PC systems. While many seemed knowledgeable and to possess the necessary technical expertise, others did not. Also, the remediation effort is off to a slow start. According to the May 1998 status report, only half of the PC-based applications have been assessed.

As stated above, OIT has many challenges, and its resources are already stretched. Nevertheless, to ensure a timely and effective remediation effort for PC systems, it will need to provide additional assistance to the divisions and offices. One possibility would be for OIT to expand the support it is currently providing through a National Institute of Health government wide contract.

Recommendation N

In consultation with the Chairman's Office and the Office of the Executive Director, OIT should seek ways to offer additional technical assistance (such as expanded contractor support) to the divisions and offices performing year 2000 remediation. It should also alert the divisions and offices to the Internet sites containing year 2000 information that could be helpful in their efforts.

Attachment 1

Five Phase Model

The five phase model approach to year 2000 remediation is briefly described below. For additional information see GAO's Year 2000 Computing Crisis: An Assessment Guide (See http://www.gao.gov) and GSA's Year 2000 Best Practices (http:www.itpolicy.gsa.gov/gsacio/bpfedgud.htm)

Awareness:

Define the Year 2000 problem and gain executive level support and sponsorship. Establish Year 2000 program team and develop an overall strategy. Ensure that everyone in the organization is fully aware of the issue.

Assessment:

Assess the Year 2000 impact on the enterprise. Identify core business areas and processes, inventory and analyze systems supporting the core business areas, and prioritize their conversion or replacement. Develop contingency plans to handle data exchange issues, lack of data, and bad data. Identify and secure the necessary resources.

Renovation:

Convert, replace, eliminate selected platforms, applications, databases, and utilities. Modify interfaces.

Validation:

Test, verify, and validate converted or replaced platforms, applications, databases, and utilities. Test the performance, functionality, and integration of converted or replaced platforms, applications, databases, utilities, and interfaces in an operational environment.

Implementation:

Implement converted or replaced platforms, applications, databases, utilities, and interfaces. Implement data exchange contingency plans, if necessary.

Attachment 2

OMB Year 2000 Target Dates ⁷

February 1998	Inventory of all outside data exchanges
March 1998	Coordinate with outside parties for a transition plan
September 1998	Renovation completed
January 1999	Validation completed
March 1999	Implementation completed
	Data exchanges fixed
	Contingency plans in place for systems not implemented

Endnotes

¹ For purposes of this report, remediation refers to the entire process in making systems year 2000 compliant; i.e., awareness, assessment, renovation, validation, and implementation.

²SEC Year 2000 Report, Future Reports Could Provide More Detailed Information, GAO (GGD/AIMD-98-51), March 1998

³The Commission's May 1998 report to OMB is attached in the Appendix.

⁴ In its May 1998 status report, the Commission indicated that EDGAR has been assessed and has been deemed compliant.

⁵ See the Appendix for an excerpt from NIST special publication 500-234 on IV & V.

⁶ As well as general experience with system development projects throughout the government and private sector.

⁷ (Source: OMB memorandum 98-02)