Skip to Main Content

Oversight of SRO Automation

This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.

OVERSIGHT OF SRO AUTOMATION

Audit Report No. 268
May 18, 1998

EXECUTIVE SUMMARY

Our review found that the Division of Market Regulation has generally implemented the Commission's Automation Review Policy (ARP) effectively. We are making several recommendations in the Audit Results section to enhance its efforts, including considering retention bonuses; enhancing training for ARP staff; and improving risk assessments.

The Division of Market Regulation and the Office of Compliance Inspections and Examinations (OCIE) submitted written comments on our draft report (attached). The Division of Market Regulation concurred with most of our recommendations. OCIE did not concur with the one recommendation addressed to it, although it supported the underlying purpose of the recommendation. We have modified the report as appropriate to reflect the comments.

SCOPE AND OBJECTIVES

Our objective was to determine whether the Division of Market Regulation implemented the Automation Review Policy program effectively. During the review, we interviewed Commission and Self-Regulatory Organization (SRO) staff and reviewed available documentation.

The audit was performed from June 1997 to January 1998 in accordance with Generally Accepted Government Auditing Standards.

BACKGROUND

The Commission issued its first Automation Review Policy (ARP) statement in 1989 (Release No. 34-27445), followed in 1991 by ARP2 (Release No. 34-29185). The ARP statements are not rules, but rather general statements of policy based on cooperation between the SROs and the Commission.

The intent of the statements is to enhance the SROs' automation capabilities and their communication with the Commission on automation issues. The statements covered SRO system capacity and assessment, notification of system outages, annual reports on systems, and independent reviews. In 1993 and 1994 the Commission adopted a risk analysis approach and broadened coverage to include clearing houses.

The first ARP statement indicated in footnote 17 that "the Commission notes that compliance with this policy statement by SROs is voluntary. The Commission's examination program, however, will review carefully the preparedness of SRO systems to handle substantial volume spikes. If the Commission becomes concerned over the level of voluntary compliance with this Policy Statement, it may propose a rule that would place an affirmative obligation on the SROs to obtain a periodic review of their automated systems. While this Policy Statement does not directly discuss the obligations of broker dealers, proprietary trading systems, service bureaus, and vendors, the Commission believes all should engage in system testing, and this Policy Statement should be used as a guideline."

In 1997 the Commission issued two concept releases (Release Nos. 34-38672, 34-38860) soliciting comments for reevaluating its approach to the regulation of exchanges and other markets in light of technological advances and the corresponding growth of alternative trading systems and cross-border trading opportunities. Under consideration with this concept release were proposals for integrating alternative trading systems into the existing broker dealer regulatory structure and for ensuring adequate capacity and the integrity of alternative trading systems. The Commission has recently acted on the concept release and published a release proposing new rules for the regulation of exchanges (Securities Exchange Act Release No. 39884, April 17, 1998).

The Division of Market Regulation implements ARP through a Technology Team (formerly a branch) headed by a Team Leader. The team members consist of computer specialists and information system auditors, each of whom is a desk officer for one or more SROs. Their duties include SRO inspections, evaluating rules, tracking system outages, reviewing SRO annual reports, and acting as liaison with SRO information system (IS) staff. The Team Leader notifies senior division staff of any significant IS problems.

AUDIT RESULTS

We found that Market Regulation has generally implemented the ARP program effectively. Because of ARP, the Commission is more knowledgeable about automation issues affecting the securities industry than it otherwise would have been. Also, some SRO staff indicated that the ARP program provides SRO management an additional impetus for implementing automation improvements.

Our recommendations to further enhance the ARP program follow.

Voluntary Status of ARP

As stated in the Background, the first ARP statement indicated that compliance by the SROs with it was voluntary. The statement further indicated that the Commission might use rule-making if the level of SRO compliance was not satisfactory. The Commission has not yet indicated how it will assess compliance. Moreover, several SRO staff indicated that they do not view the program as voluntary, given the possibility of rule-making.

Given the ever increasing importance of automation issues, the Commission may want to assess whether ARP should remain a voluntary program or become mandatory (that is, through rule-making). A senior official in the Division of Market Regulation indicated that the ARP program will remain voluntary for the time being. However, this issue is being further considered in Concept Release 34-38672 (see Background).

Recommendation A

The Division of Market Regulation should reconsider whether to recommend that ARP remain a voluntary program, or become mandatory through rule-making. The Division should also decide how it will assess SRO compliance with ARP.

Staff Turnover

The Market Regulation Technology Team has experienced significant staff turnover within the last two years, as three technology staff have left the Commission during that period. New staff are not as familiar with the operations of the SROs and clearing houses, which can adversely effect the efficiency and effectiveness of the ARP program. Moreover, new staff may not fully understand risks associated with system changes.

The Division has indicated that it does not send inexperienced staff into the field unless they are accompanied by more experienced staff and that the staff use materials from past reviews to prepare for on-site visits. However, SRO staff indicated that the SROs have spent time educating Commission staff during their annual reviews, although they indicated that this problem has improved lately.

The Commission has proposed awarding retention bonuses to selected staff. These bonuses could help reduce turnover on the Technology Team.

Recommendation B

If the Commission implements a program of retention bonuses, the Division of Market Regulation should consider offering bonuses to selected members of the Technology Team.

Training

The automation issues in the ARP program are technical and rapidly changing. Also, each SRO has its own hardware and software platforms. Training is consequently important for the Technology Team, especially in view of the staff turnover discussed above.

We reviewed training records for the technology staff. We compared their training to the subject areas of ARP and to the SROs' hardware and software platforms (as listed in a profile guide prepared by Market Regulation). Each staff member took one to three courses per year. For the most part, the training appeared general rather than specialized. In addition, coverage of the ARP areas and the SROs' platforms was limited. However, a recent training request from the Technology Team proposed increased training over the next few years.

Recommendation C

The Division of Market Regulation should ensure that technology staff have training in all ARP areas, especially the hardware and software platforms used by SROs.

Recommendation Tracking System

The Technology Team developed a tracking system using Microsoft Access to record all recommendations to SROs on automation issues. The recommendations come from SROs' internal and external auditors, the General Accounting Office, and the Commission. The system contains the source of the recommendation, a brief description, the issued and response dates, and the current status (open, closed, or disagree). The recommendations in the system date from 1991.

The Technology Team could analyze this data to identify trends and to assist its planning and risk assessment. For example, we developed two schedules from this data. One shows the total number of recommendations for each SRO with the number and percentage of open recommendations. The other is an aging distribution. It shows the number and percentage of open recommendations for each six month interval and cumulative percentage totals. Using these schedules, we noted that more than half of all open recommendations were more than three years old. Also, some SROs appeared to be less active in resolving recommendations than others.

Recommendation D

The Division of Market Regulation should analyze data from its recommendation tracking records for risk management and evaluating compliance with ARP. It should share this analysis, as appropriate, with the Office of Compliance Inspections and Examinations.

Risk Assessments

Desk Officers prepare risk assessments of the SROs, which the Supervisory Computer Specialist uses in planning SRO inspections. The risk assessments include several categories, such as computer operations, telecommunications, data security, system development methodology, capacity, contingency, internal audit, risk analysis performed by SROs, application systems, annual reports, and system changes and outages. The desk officer uses the analysis in the categories to assist in rating overall SRO risk as high, medium, or low.

Several categories (contingency, application systems, annual reports, and SRO risk analysis) are used by some Desk Officers and not others. Also, depending on the type of entity (e.g., exchange, clearing house, or depository) and systems involved, some categories may be less important than others. For example, system outages may not be as important for batch processed systems as for on-line systems. Finally, the ARP statements define only some of the risk assessment categories used by the Desk Officers.

Recommendation E

The Division of Market Regulation should issue guidance on risk assessments of SROs to its technology staff. The guidance should define categories not already defined by the ARP, standardize categories among desk officers, and describe circumstances when categories should be given less weight.

Attorney Involvement with ARP

The Technology Team is composed of computer specialists and information systems auditors, while the rest of Market Regulation's staff consists mostly of attorneys without specialized computer training.

To ensure that ARP issues are fully understood and addressed, additional involvement in the ARP program by Market Regulation attorneys appears desirable. SROs or Technology Team staff may be willing to make presentations at headquarters to Market Regulation and other Commission staff. Or, attorneys could visit SROs (e.g., to obtain a tour of data processing facilities and participate in Technology Team activities).

A senior Market Regulation official indicated that attorneys are already involved in the ARP program to some extent (e.g., attorneys have participated in ARP reviews).

Recommendation F

The Division of Market Regulation should provide opportunities for increasing the knowledge and involvement of attorneys in the ARP program.

Comments on ARP reviews

The Division of Market Regulation's policy is not to issue draft reports to SROs for comment before they are made final, although it generally provides the SRO with an exit conference. To ensure that the SRO's perspective is fully understood, the Division should make exit conferences mandatory, unless senior management makes an exception.

Recommendation G

The Division of Market Regulation should require exit conferences to be held for all ARP reviews, including cause examinations. The exit conference should be documented by the technology staff, including any comments of the SRO.

Availability of ARP Statements

The Commission intends that the ARP statements will provide guidance to the securities industry, including SROs, broker dealers, proprietary trading systems, service bureaus, and vendors. This intent can be promoted by making the ARP statements widely available on the Internet.

Recommendation H

The Division of Market Regulation should post the ARP statements on the Commission's Internet web site.

Automation Industry Reference Materials

The Technology Team has established its own library consisting of books, periodicals, and other resource materials on automation issues. This library could be enhanced by obtaining comprehensive computer industry reference materials covering the availability of capacity modeling tools, software change management tools, hardware and software features of various systems, and security, among other areas. The library does not contain, for example, the complete DATAPRO series, an important reference work on the automation industry.

Recommendation I

The Division of Market Regulation should provide comprehensive automation industry reference materials to the Technology Team. As one example, the entire DATAPRO series on CD-ROM could be made available.