This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.

GPRA PERFORMANCE MEASURES

AUDIT NO. 283
August 24, 1998

EXECUTIVE SUMMARY

We found that the Commission can improve the support for performance measures in its 1999 performance plan under the Government Performance and Results Act (GPRA). Half of the measures we reviewed (8 of 16) did not agree with supporting records. Like other federal agencies, the Commission apparently encountered difficulties in the definition, data collection, and reporting phases of performance measurement.

Besides recommendations to improve the supporting records, we are recommending additional GPRA training and guidance, and improved controls in the tracking systems generating the measures. Commission offices and divisions have already taken several corrective actions in response to our findings and recommendations, as noted below.

OBJECTIVES, SCOPE, AND METHODOLOGY

Our objective was to determine whether fiscal year 1997 performance measures in the 1999 performance plan materially agreed with supporting records. We wanted to help ensure that reported data are adequately supported when the Commission makes its first performance report under GPRA (due by March 31, 2000).

We selected a judgment sample of 16 of 27 output performance measures in the FY 1999 performance plan (some measures included multiple individual measures), and examined supporting records, including computer print-outs and source documents. Output (rather than outcome) measures were selected for review because they represented actual rather than projected activity, were specifically related to the Commission's strategic plan, and generally included targets. We did not evaluate whether the measures themselves were appropriate.

A previous audit (Audit No. 272) made recommendations concerning the tracking system for one of the 16 measures initially selected (Number of Commission and self-regulatory organization (SRO) broker-dealer investor protection rules reviewed by the Division of Market Regulation). The Division is currently addressing these recommendations.

We attempted to recreate each selected measure using the records provided. The Appendix lists the specific performance measures reviewed. In addition, we interviewed Commission personnel and conducted limited testing of the tracking system data upon which the measures were based. The audit was performed from April to August 1998 in accordance with generally accepted government auditing standards.

BACKGROUND

The Government Performance and Results Act of 1993 requires agencies to define their mission and goals, measure their performance toward those goals, and report to the Congress and the public on their progress. Agencies must develop five-year strategic plans, annual performance plans and annual performance reports.

The first performance plan is for fiscal year 1999 and the first performance report is due no later than March 2000. Performance plans and reports should include quantitative performance measures, unless the agency obtains OMB approval for alternative measures.

In the future, agencies will link their budget requests to their GPRA plan. The linkage between the performance plan and the budget will be based on the program structure of the agency. Agencies will define their performance goals and measures based on their program activities.

The Commission organized a GPRA Task Force, composed of senior staff from the offices and divisions, to coordinate its efforts. The Chairman's Office, the Office of the Executive Director, and the Office of the Comptroller share overall responsibility for implementing GPRA requirements. These offices also oversee the Commission's proposed budget.

As a result of the Task Force's efforts, the Commission has generally complied with GPRA requirements. It issued a five-year strategic plan in September 1997, and annual performance plans for 1999 and 2000.

AUDIT RESULTS

We found that the Commission could improve the supporting records for the FY 1997 performance measures. While some measures materially agreed (within 5%) with supporting records, others did not, for a variety of reasons, including unclear definitions of measures and incomplete records.

The Commission appears to have encountered many of the same obstacles as other Federal agencies in defining, collecting information for, and reporting performance measures. The General Accounting Office described these difficulties in two recent reports.

A detailed description of our findings and recommendations follows. See the Appendix for an overall summary of our results.

MEASURES AGREEING WITH SUPPORTING RECORDS

• Number/percent of enforcement actions by program (Total)(Division of Enforcement)
• Number of SEC requests to foreign regulators for enforcement assistance (Office of International Affairs)
• Number of requests from foreign regulators to SEC for enforcement assistance (Office of International Affairs)
• Average number of files downloaded per day - SEC web site (Office of Information Technology)
• Number/percent of new mutual fund and closed-end fund portfolios reviewed (Division of Investment Management)
• Percent of examinations of broker-dealers resulting in significant findings (Office of Compliance Inspections and Examinations)
• Complete all planned SRO regulatory inspections (Percentage completed) (Office of Compliance Inspections and Examinations)
• Number/Percent of formal responses to requests for interpretive advice and exemptive relief (Division of Market Regulation)

MEASURES NOT QUANTIFIED

• Percent of investment company complexes examined (Office of Compliance Inspections and Examinations)

Inspections of investment companies were not quantified for FY 1997 because the five year plan was not completed until the middle or end of FY 1997. The five-year plan developed for these examinations included information from FY 1998 forward.

• Percent of investment advisers examined (Office of Compliance Inspections and Examinations)

Investment advisers were not included for FY 1997 because the states were going to take over the examinations for some of the smaller advisers. As a result, FY 1997 data would not have been comparable to FY 1998 and forward. Since FY 1999 is the first year for which the GPRA requires performance measures, the Office of Compliance Inspections and Examinations' explanation appears reasonable.

• Rulemaking to modernize and streamline securities offerings rules (Division of Corporation Finance)

The Division of Corporation Finance stated that it had difficulty developing meaningful numbers for rulemaking to modernize and streamline securities offering rules. It is considering whether this measure should be retained.

In 1996, the Commission's Task Force on Disclosure Simplification recommended elimination of 81 rules and 22 forms. By the beginning of FY 1998, the Commission had eliminated approximately 46 rules and six forms. This kind of information could be used to quantify the Division of Corporation Finance's measure.

Recommendation A

The Division of Corporation Finance should either develop a method to quantify its rulemaking measure, obtain OMB approval for an alternative form of measurement, or discontinue the measure.

MEASURES NEEDING BETTER DEFINITIONS

• Number of initial public offerings, reporting issuers filings reviewed (Division of Corporation Finance)

Through an oversight, the Division of Corporation Finance did not include reviews of new reporting issuers (under the 1934 Securities Exchange Act) in the definition of the number of initial public offerings and reporting issuer filings.

• Number/percent of new insurance contracts reviewed (Division of Investment Management)

This measure was an estimate. Rather than counting the number of insurance contract filings reviewed, the Office of Insurance Products in the Division of Investment Management assumed that this number equaled the number of filings received.

• Number/percent of formal responses to requests for exemptive relief (Division of Investment Management)

The data comprising this measure were inconsistent. Two offices within the Division of Investment Management process requests for exemptive relief (Insurance Products and Investment Company Regulation). The Office of Investment Company Regulation included requests withdrawn and requests declared inactive in their total of formal responses, while the Office of Insurance Products did not.

Recommendation B

The Divisions of Corporation Finance and Investment Management should clarify the definition of the measures discussed above, and correct the data in the performance plan as appropriate.

Management response: The Division of Corporation Finance has already completed action on this recommendation. The Division of Investment Management will develop uniform definitions for reviewed filings and responses to requests for exemptive relief.

INCOMPLETE SUPPORTING RECORDS

• Percent of risk assessment reports resulting in follow-up (Division of Market Regulation)

This percentage was an estimate. Risk assessment staff sometimes followed-up orally on the reports and did not always input the information into the designated tracking system.

• Number of market regulation matters reviewed for the Division of Enforcement (Division of Market Regulation)

Division of Market Regulation staff told us that part of the manual logbook used to record market regulation matters reviewed for the Division of Enforcement was misplaced during a recent office move.

• Number of foreign private issuers registering under the 1933 and 1934 Acts for the first time (Division of Corporation Finance)

• Dollar amount of securities registered for sale under the 1933 Act by all foreign issuers (Division of Corporation Finance)

Corporation Finance was unable to locate supporting records and computations for new foreign issuers and the value of registered securities by foreign issuers. The summary numbers it provided substantially exceeded the numbers in the performance plan (203 new foreign issuers for FY 1997 per supporting data versus 75 in the performance plan, and $93 billion in securities registered in FY 1997 by all foreign issuers per supporting data versus $40 billion in the performance plan). A Division of Corporation Finance official suggested that Canadian filers may have been left out of the performance plan data.

Recommendation C

The Divisions of Market Regulation and Corporation Finance should develop procedures (e.g., back-ups of data sources, assignment of responsibilities, periodic spot checks) to ensure that records are available to support performance measure data.

rounding error

• Dollar value of small business registrations filed

The Division of Corporation Finance rounded $6.69 billion to $6 billion, when it should have reported either the exact amount, or a rounded total of $7 billion. The Division of Corporation Finance has already corrected this error.

gpra training and guidance

Several managers we interviewed who were responsible for GPRA measures appeared unfamiliar with GPRA requirements, including performance measurement. GPRA guidance and training would be helpful to Commission staff as the Act is implemented.

Some offices have already taken steps to increase staff understanding of the Act. For example, the Division of Corporation Finance, the Office of Compliance Inspections and Examinations, and the Office of International Affairs independently researched methods for developing improved performance measures. Also, the Comptroller's office discussed the Act in a 1996 column in the Employee News . We commend their efforts.

The Comptroller, as head of the GPRA Task Force, can build on these efforts by issuing guidance and identifying appropriate training.

Recommendation D

The Comptroller, in coordination with the Executive Director, should issue guidance to Commission managers which briefly describes the GPRA requirements and their relevance to Commission operations. She should also include information on sources of training in the GPRA requirements and developing performance measures.

tracking systems for gpra data

Of the 16 measures we selected, at least 9 were based on data from user-developed systems, using software such as Access, Foxpro, Dbase, or Q&A. The systems were not developed specifically to record GPRA performance measure data, and had certain control and documentation deficiencies.

The extent of internal controls varied widely, depending on the system's intended use. For example, IM's filing review tracking system, for division-wide use, included controls over access to the system data, edit controls, and an error report generation capability. On the other hand, Corporation Finance's database of foreign issuers, developed for an individual's use, had few controls.

Two tracking systems, in OCIE and Enforcement, are scheduled to be replaced. The new systems are expected to include improved controls.

Investment Management provided input instructions for its filing review tracking system. However, none of the systems had sufficient documentation (e.g., user manuals) to allow a new system administrator to operate them.

These internal control deficiencies did not appear to cause the lack of quantification, unclear definitions, incomplete supporting records, and rounding error noted in this report. However, the staff responsible for these tracking systems should improve the internal controls, especially since GPRA data must be provided to OMB and the Congress.

Recommendation E

The Divisions of Corporation Finance and Investment Management should ensure that their systems that maintain GPRA data include adequate internal controls. The Office of Information Technology should assist the staff in identifying and implementing the appropriate system controls.