Compliance Inspections and Examinations
March 18, 1998
Audit No. 254
This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.
COMPLIANCE INSPECTIONS AND EXAMINATIONS
Audit Report No. 254
March 18, 1998
EXECUTIVE SUMMARY
The Securities and Exchange Commission's Office of Inspector General (OIG) conducted an audit of the Commission's compliance inspections and examinations function (examination function). Audit workshops were conducted at headquarters and at six regional and district offices.
Participants believed that, taken as a whole, the examination function is generally achieving its primary objective to conduct effective examinations and inspections of regulated entities to protect investors. They believed that most supporting objectives are generally being implemented, although some obstacles exist which are impairing full implementation. However, two supporting objectives (staff training and staff retention) were rated 3.9 and 3.5, respectively, on a scale of 1.0 through 7.0, where 3.0 meant that the supporting objective is generally not being implemented (many obstacles are impairing full implementation) and 5.0 meant that the supporting objective is generally being implemented (some obstacles are impairing full implementation). The rating scale used is presented as Appendix B. Participants overwhelmingly reported that performing effective and efficient examinations is a high priority within the examination function.
A significant finding concerned the examiner training program. Specifically, participants felt that the Office of Compliance Inspections and Examinations (OCIE), which directs the examination function, should introduce additional training topics, while expanding the coverage of others. To their credit, OCIE management demonstrated a strong commitment to providing training to its examiners, evidenced by recently arranging for a contractor to provide periodic and comprehensive training program for its examiners. 1 Such training is essential and necessary in conducting effective examinations.
Staff retention was identified as a significant obstacle to achieving the examination function's objectives. Participants indicated that recent significant staff turnover has had a negative effect on productivity and morale. We understand the legal and political constraints facing the Commission in this area, but identified several improvements for consideration.
Information systems support was also a concern raised by management and participants. The examination function is in need of additional computer support to maintain and improve its examinations. Specifically, participants identified several systems in need of modification, expanded examiner access to the "Internet", and examiner assistance in downloading data and conducting database searches. Technology has become an integral part of the examination process.
The participants in the workshops also expressed a desire for better communication of goals, objectives and examination information between OCIE headquarters and the field offices, improved examination targeting and follow-up efforts, and enhanced accountability for managers and staff.
The examination function was chosen for evaluation because performing compliance inspections and examinations is critical to the Commission's overall goal of investor protection. Our review of the examination function was included in our 1997 audit plan and was not prompted by complaints.
The aim of the critique was to draw upon the knowledge and experience of staff and managers through a series of structured workshops. A recognized internal audit methodology (Control Self Assessment or CSA) was used to systematically collect this information. Working with OIG staff, management developed the primary and supporting objectives for the examination function. Personal assessments from OCIE and field office professional staff and managers of how successfully the examination function achieves its objectives were obtained.
BACKGROUND
The Commission's examination function was consolidated in May 1995 to create OCIE. OCIE was designed to streamline and improve the examination process. As part of the Commission's broad mission, OCIE's mandate is to protect investors through fostering compliance with the securities laws, detecting violative conduct, and ensuring that the Commission is informed of developments in the regulated community.
There are three primary examination areas: Broker/Dealers (B/D), Investment Advisers/Investment Companies (IA/IC) and Self-Regulatory Organizations (SRO). OCIE exercises program supervision over the examination function in the Commission's eleven regional and district offices across the United States. In addition to actually performing examinations, OCIE staff determine examination policy and procedure.
The examination function includes approximately 600 Commission personnel. In the regional offices, about 200 staff work in the B/D area and about 320 work in the IA/IC area. About 80 work in OCIE's Washington, DC headquarters, which includes staffing for the SRO area. The Commission examines approximately 650 B/D's, 300 IC complexes, 1,200-1,600 2 IA's, and 40 SRO programs annually.
SCOPE AND OBJECTIVES
The scope of our audit included conducting structured workshops with staff and management from OCIE's three main examination areas (i.e., B/D, IA/IC, SRO). Staff and managers from six regional and district offices also participated in workshops. In all, 207 participants out of approximately 600 (representing 35%) of those working in the examination function took part in assessing the achievement of objectives.
The primary objective of the audit was to determine the extent to which the examination function is achieving its primary and supporting objectives. The review also recommended actions to increase the likelihood that those objectives would be achieved.
During the audit, OCIE management documented the primary objective of the examination function:
Conduct Effective Examinations and Inspections of Regulated Entities to Protect Investors
Eight supporting objectives, reflecting the activities that make achievement of the primary objective more likely, were also developed by management. They were:
TARGETING EXAMINATIONS / INSPECTIONS - Select regulated entities for review in order to achieve strategic targeting goals (e.g., risk, routine cycle, oversight).
EXAMINATION / INSPECTION EFFECTIVENESS - Ensure that examinations and inspections are effective in uncovering significant non-compliance with securities laws, or weaknesses in oversight.
EXAMINATION / INSPECTION FOLLOW-UP - Ensure that regulated entities appropriately address examination/inspection findings in a timely manner.
EXAMINATION / INSPECTION EFFICIENCY - Ensure that resources are efficiently utilized and management controls are effectively implemented.
STAFF TRAINING - Ensure that staff have sufficient training to perform effectively.
COMMUNICATIONS - Keep staff informed of industry trends, and Commission policies and positions.
PERFORMANCE MEASURES - Measure and evaluate individual performance in conformity with program objectives and goals.
STAFF RETENTION - Promote a work environment and career track that fosters the retention of qualified employees. The field work was performed from November 1996 through December 1997 in accordance with generally accepted government auditing standards.
AUDIT METHODOLOGY
A major goal of the audit was to maximize the value of the audit to management, while maintaining auditor independence. A version of a private sector, internal audit methodology (Control Self-Assessment or CSA) was adapted for this purpose. The Institute of Internal Auditors has promoted the concept internationally for the last several years with outstanding results reported.
The premises for using this approach are common to many applications that use a team approach to decision making (e.g., teams can process more diverse information than an individual, employees are closer to the information needed than anyone else, staff can make a good system fail or vice versa, etc.).
Primary data collection was accomplished through 16 workshops (207 employees or approximately 35% of examination function personnel participated in the workshops) in which the supporting objectives were discussed. Participant ratings of how well the staff achieved the primary and supporting objectives, as well as the importance of each of the eight supporting objectives, were collected. No document reviews or other audit tests were performed. The methodology provided perceptions and judgments about the success of examination function objectives.
The CSA methodology used in the audit consists of four primary tasks: identify management objectives, convene workshops to discuss and rate each objective, evaluate the workshop data, and prepare an audit report. These audit steps are described in further detail in Appendix A.
A number of issues were raised in the workshops that were only tangentially related to compliance inspections and examinations. These matters were referred to appropriate management and may form the basis for future OIG work.
AUDIT RESULTS
Workshop participants anonymously rated how well the examination function achieved its primary objective, as well as each of its eight supporting objectives. The rating scale used by the participants ranged from 7.0 (full implementation) to 1.0 (not being implemented in a meaningful manner). The actual effectiveness and importance scales used are included in Appendix B. The overall rating (all workshop participants) for how well the examination function is actually achieving its primary objective to conduct effective examinations and inspections of regulated entities to protect investors was 5.3.
The composite ratings (all workshop participants) for how well the examination function actually achieved its eight supporting objectives were as follows:
|
5.0 |
|
5.0 |
|
4.6 |
|
4.6 |
|
3.9 3 |
|
4.3 |
|
4.3 |
|
3.5 |
The composite ratings of actual achievement by the participating staff indicate that most supporting objectives are generally being implemented, although some obstacles exist which are impairing their full implementation. However, they rated two supporting objectives (staff training and staff retention) as 3.9 and 3.5, respectively, on a scale of 1.0 through 7.0, where 3.0 meant that the supporting objective is generally not being implemented (many obstacles are impairing full implementation) and 5.0 meant that the supporting objective is generally being implemented (some obstacles are impairing full implementation). The rating scale used is presented as Appendix B.
Participants felt that OCIE should introduce additional training topics, while expanding the coverage of others. To their credit, OCIE management demonstrated a strong commitment to providing training to its examiners, evidenced by recently arranging for a contractor to provide periodic and comprehensive training program for its examiners.4 Such training is essential and necessary in conducting effective examinations.
Staff retention was identified as a significant obstacle to conducting effective and efficient examinations. Participants indicated that recent significant staff turnover has had a negative effect on productivity and morale. We understand the legal and political constraints facing the Commission in this area, but identified several improvements for consideration.
Information systems support was also a concern raised by participants. The examination function is in need of additional computer support to maintain and improve its examinations. Specifically, participants identified several systems in need of modification, expanded examiner access to the "Internet", and examiner assistance in downloading data and conducting database searches. Technology is an integral part of the examination process.
The participants in the workshops also expressed a desire for better communication of goals, objectives and examination information between OCIE headquarters and the field offices, improved examination targeting and follow-up efforts, and enhanced accountability for managers and staff.
Although managers consistently rated the achievement of objectives higher than did the staff participants, the ratings of the two groups were reasonably aligned. The staff ratings of the relative importance of the supporting objectives were generally similar to the ratings of the manager workshops and, generally, the relative trends of their ratings were directly correlated.
TARGETING EXAMINATIONS / INSPECTIONS
Both actual achievement and level of importance of the targeting of examinations objective was rated among the highest of the eight supporting objectives. Participants from all three examination areas agreed that targeting should largely be based on assessed risk. IA/IC and SRO participants generally felt that they were given adequate time to perform quality examinations. B/D participants were more mixed regarding being afforded sufficient time, but indicated that an adequate number of staff is usually assigned to examinations.
Investment Adviser/Investment Company managers expressed concern that they do not receive complaints about firms received at headquarters. Based on participant comments, OCIE management discussed the situation with the Office of Investor Education and Assistance (OIEA). It turned out that this information is available on-line to all IA/IC examiners through their local OIEA liaison.
Although B/D and IA/IC examiners in some offices indicated that they provide input to their supervisors for targeting, others stated that they are not involved in the targeting process. Examiner observations could prove useful in the targeting process. Accordingly, participants felt that an additional avenue for examiners to provide targeting input to managers would be beneficial.
Participants also felt that OCIE should regularly inform all SRO's about best practices observed by Commission staff during its oversight. These observations identify potential lessons learned and can be communicated to applicable SRO's, which would help protect investors.
Recommendation A
The Office of Compliance Inspections and Examinations should provide periodic reminders to IA/IC examiners in field offices that complaint information received at headquarters is available to them and how they can obtain it in a timely manner.
Recommendation B
The Office of Compliance Inspections and Examinations should consider establishing a policy of giving B/D and IA/IC examiners the option of including a recommendation in the workpapers as to when a subsequent examination of a firm may be appropriate and its scope.
Recommendation C
The Office of Compliance Inspections and Examinations should regularly inform SROs about best practices observed by Commission staff during its oversight.
The Office of Compliance Inspections and Examinations management informed us that, over the last few months, OCIE and regional office staff have met with state and SRO examiners to coordinate examinations and share information about each others' programs. In addition, OCIE management said that it has begun providing SRO's with more information about "best practices" that Commission staff has observed during inspections.
EXAMINATION / INSPECTION EFFECTIVENESS
The achievement and importance of the examination effectiveness objective was rated among the highest of the eight supporting objectives. Several key factors were raised as important elements of examination effectiveness. B/D participants generally felt that pre-examination planning was vital. IA/IC participants felt that the levels of experience and knowledge of examiners was important. Additionally, SRO participants saw great value in the current OCIE practice of interviewing SRO staff during examinations, instead of merely performing document reviews.
A key area of concern among participants related to information technology. IA/IC participants stated that recent improvements in automation have helped examination effectiveness, but further improvements were needed. For example, participants wanted the NSAR system modified so that each "series" belonging to an investment company is identified with a unique number. In addition, the participants complained that the information contained in the NRSI, Workload, and Matter Under Investigation (MUI) systems was not always accurate nor timely.
Investment Adviser/Investment Company examiners felt that acquiring or expanding access to pricing services and informational resources in the field offices would aid them in planning and conducting examinations. Pricing services mentioned as desirable included Standard & Poor's, Muller, and Reuter's. Informational services recommended included Nelson's Directory of Investment Managers, Bloomberg, Morningstar, and Lipper.
Broker/Dealer and SRO participants felt that inspection staff should have access to the Internet in the field to facilitate more effective examinations. We understand from the Office of Information Technology (OIT) that there are certain legal and privacy concerns with granting remote access to the "Internet" over the Commission's network. However, given the increasing amount of information available on the "Internet", access (even from an independent service provider) would greatly enhance examination effectiveness.
The Office of Compliance Inspections and Examinations has continued its efforts to keep examiners informed of policy determinations by issuing a periodic newsletter, as well as issuing policy statements and updating procedures (e.g., B/D guidelines were updated in November 1996). B/D and IA/IC participants appreciated the value in such efforts, but desired additional guidance in certain areas. For example, participants reported that they were reluctant to narrow the scope of an examination during "smart" examinations for fear of missing a problem. These participants also wanted more policy/procedural information on "sweep" examinations, as well as whether or not to conduct exit interviews.
Participants in each of the three examination areas identified numerous "best practices" that would add to examination effectiveness. Although OCIE management felt that many of the practices were already in place, after reviewing participant comments, they felt that preparing and issuing a "best practices" document to examination staff would be useful, especially for new staff. Suggestions from workshop participants (including the applicable examination areas) are included in Appendix C.
Recommendation D
The Division of Investment Management, in coordination with OCIE, OIT and the Office of Filings and Information Services (OFIS), should consider modifying the NSAR system so that each "series" belonging to an investment company is identified with a unique number.
Officials from IM informed us that the request for proposal for the current EDGAR system modernization project addresses the capability of the system to identify a specific series belonging to an investment company.
Recommendation E
The Office of Compliance Inspections and Examinations should formally notify the Commission's Information Technology Capital Planning Committee and OIT of the problems and consequences of inaccurate and untimely data in specific Commission databases.
Recommendation F
The Office of Compliance Inspections and Examinations, in conjunction with OIT, should consider providing examination staff with expanded access to the Internet (including the possibility of establishing remote accounts through commercial service providers) for B/D and IA/IC examiners in the field.
Recommendation G
The Office of Compliance Inspections and Examinations, in coordination with the Office of the Secretary, the Executive Director and the External Database Steering Committee, should consider the feasibility of providing or expanding access to the pricing services and informational resources mentioned above to aid IA/IC examiners in planning and conducting examinations.
Recommendation H
The Office of Compliance Inspections and Examinations should provide additional and more detailed guidance to B/D and IA/IC examiners on "smart" examinations, "sweep" examinations, and holding exit interviews.
Recommendation I
The Office of Compliance Inspections and Examinations should prepare and issue a best practices document to all examination staff. OCIE should consider participant recommendations listed in Appendix C.
EXAMINATION / INSPECTION FOLLOW-UP
Overall, participants responded favorably to examination follow-up activities. OCIE management participants gave follow-up the highest achievement rating of all the supporting objectives.
Investment Adviser/Investment Company participants felt that exit interviews were effective from the perspective of both the examiner and the regulated entity. They also indicated that referrals to the Division of Enforcement (Enforcement) were an effective means of addressing serious problems. Participants believed that, in most cases, follow-up examinations showed that deficiencies cited in earlier examinations were corrected. Finally, B/D and IA/IC participants felt that deficiency letters are effective in getting corrective action on most deficiencies identified.
A major issue among all participants was Enforcement referrals. Participants from all three examination areas were concerned that they were not kept informed of the status of referrals. Some B/D and IA/IC participants felt that some important deficiencies were referred to Enforcement and not included in deficiency letters and hence were "dropped," if Enforcement declined to pursue a matter. In addition, participants wanted more guidance on Enforcement's criteria for a successful referral. Finally, IA/IC participants stated that the IRMS system needed to be expanded so that users can track the date and nature of the deficiency, as well as any follow-up actions taken.
Broker/Dealer participants expressed concern that not all firms send responses to Commission deficiency letters to their respective SRO. Participants felt that it would be useful if firms were generally required to forward this information to the SRO.
Recommendation J
The Office of Compliance Inspections and Examinations, in coordination with Enforcement and OIT, should ensure that examiners are regularly provided information about the status of their Enforcement referrals. In addition, examiners should be able to efficiently track the date and nature of a deficiency, as well as any follow-up actions taken by Commission staff.
The Office of Compliance Inspections and Examinations management informed us that, following the audit workshops, OCIE began to develop a comprehensive computerized tracking system for all SEC examinations. OCIE management said that a design requirement of the system is the ability to track deficiencies and the resolution of Enforcement referrals. OCIE management further believes that this new tracking system should provide assurances that no deficiencies are "dropped".
Recommendation K
The Office of Compliance Inspections and Examinations should consult with the Office of General Counsel and Enforcement to consider whether there could be an additional means (besides current methods, such as deficiency letters) of addressing serious examination violations that do not meet the criteria for a successful Enforcement referral.
Recommendation L
The Division of Enforcement should clearly define and communicate referral criteria to examination staff.
Recommendation M
The Office of Compliance Inspections and Examinations, should take steps to require B/D's, in the appropriate cases, to provide copies of their deficiency letter responses to their respective SRO's.
EXAMINATION / INSPECTION EFFICIENCY
Participants felt that management controls in place for efficiency were generally effective. The most significant efficiency concern raised was the lack of information technology assistance within the examination function. For example IA/IC and SRO participants indicated that they needed aid in formatting downloaded data from portfolio management software. Information technology concerns were also expressed under the effectiveness objective (discussed above). For example, we noted that workshop participants desired modifications to the NSAR system, as well as expanded Internet access for examiners.
Investment Adviser/Investment Company participants expressed concern that microfiche containing information from Form ADV was not being distributed to IA/IC examiners in a timely manner.
Recommendation N
The Office of Compliance Inspections and Examinations, in coordination with OIT and the Executive Director, should consider hiring a computer consultant to assist in meeting various information technology needs of the examination function.
The Office of Compliance Inspections and Examinations management informed us that, after the audit workshops, OCIE with OIT, began a program of retaining computer consultants to assist in developing responses to the examination program's information technology needs. OCIE management said that a consultant is currently developing an examination tracking system. Additionally, OCIE has identified additional priority needs, which OIT has indicated will be dealt with in similar fashion.
Recommendation O
The Office of Compliance Inspections and Examinations, in conjunction with OFIS and the Executive Director, should improve the timeliness of distributing microfiche containing Form ADV information to IA/IC examiners.
The Office of Compliance Inspections and Examinations management informed us that, after the audit workshops, a program was launched by IM, in conjunction with state regulators, to develop an on-line data base of Form ADV information. This project is currently in the early development stage. OCIE management indicated that, once available, this should resolve current problems with using microfiche. OCIE management also indicated that the timeliness of delivering Form ADV information has improved in recent months.
STAFF TRAINING
Workshop participants generally indicated that examination function training had improved significantly in the last two years (e.g., more practical exercises). Our observations of the training efforts indicate that OCIE is committed to enhancing the effectiveness of their training programs.
Participants overall rated the achievement of staff training second lowest of the eight supporting objectives.5 However, achievement ratings obtained from the workshops could not reflect many of the improvements OCIE has made in training its examiners since the workshops were held. For example, OCIE has indicated that it is dedicated to providing periodic and comprehensive courses to its examiners. Specifically, OCIE recently began contracting with the New York Institute of Finance to provide the first year of this training.
Participants generally agreed that on-the-job training was the most valuable type of training. However, they expressed a desire to receive more formal training in specific areas. Participants in each of the three examination areas requested more training on specific issues, such as new investment/financial products (e.g., derivatives), using databases (e.g., conducting searches on Lexis-Nexis, formatting downloaded data), and selling/trading securities.
Broker/Dealer and IA/IC participants requested additional training on investigative techniques, such as fraud detection. B/D participants also requested training on the books/records of clearing firms, transfer agent operations, as well as general workpaper organization and preparation. IA/IC participants requested training on specific portfolio management software (e.g., Advent, Monarch), market making, and Microsoft Office software.
Office of Compliance Inspections and Examination management indicated that it may be more cost-effective to purchase video tapes of certain training events (e.g., industry conferences, presentations by product experts from large B/D firms) instead of sending examiners to the conferences themselves. This alternative may result in cost savings for the examination function.
Self-Regulatory Organization examiner participants rated achievement of the training objective lowest among the three examination areas and raised specific suggestions. For example, the examination function does not currently have a formal orientation program for new SRO examiners. Participants indicated that such a program is needed to expose new staff to the inspection process, as well as OCIE procedures.
In addition, participants saw value in examiners attending certain SRO training events and felt that OCIE should explore this possibility.
Recommendation P
The Office of Compliance Inspections and Examinations should consider the suggestions of the participants, discussed above, with respect to training and decide, if additional training would be cost effective, given recent improvements. If appropriate, OCIE should request additional training funds from the Executive Director.
Recommendation Q
The Office of Compliance Inspections and Examinations should consider alternate cost effective means (e.g., purchasing video tapes of certain training events) of providing necessary training.
Recommendation R
The Office of Compliance Inspections and Examinations should develop and implement a formal orientation program for new SRO inspection staff.
Recommendation S
The Office of Compliance Inspections and Examinations should request that SRO's allow SEC examiners to attend SRO training events, as appropriate.
The Office of Compliance Inspections and Examinations management indicated that, subsequent to the audit workshops, they have begun the practice of sending SEC examiners to SRO training events.
COMMUNICATIONS
Participant comments overall were positive regarding the communications objective. For example, IA/IC participants stated that they receive most communications from management through electronic mail, which has proven to be an effective medium. They agreed that closer communication between OCIE and the field offices would benefit the examination function. B/D examiners had favorable comments regarding OCIE's periodic newsletter to the staff. Additionally, OCIE management has taken steps to improve this communication. For example, subsequent to our workshops, OCIE hired two additional attorneys to address field office inquiries.
Comments about communications from B/D participants were mixed. For example, some felt that policies and procedures are communicated clearly and in a timely manner. Additionally, they felt that, overall, information is more accessible than in the past. However, some B/D participants stated that they sometimes learn of new OCIE policies from the business press, as opposed to communication from OCIE.
Workshop participants from each of the three examination areas stated that OCIE's mission, as well as changes in its goals and objectives are not being communicated to staff on a regular basis. This was viewed as a significant obstacle to achieving examination function objectives.
Some examiners were not aware that a mission statement existed and thought it was important to communicate the examination function's mission, goals and objectives to the staff at the beginning of each year.
Broker/Dealer examiners stated that an electronic bulletin board could facilitate communication among examiners in different offices. Such a bulletin board could be used by OCIE management to communicate examination guidance and for examiners to ask questions of OCIE. In reviewing workshop findings, OCIE management indicated that the examination function would be represented on the Commission's "Intranet". The "Intranet", an on-line system similar to the "Internet" but only available to Commission employees, appears to have the capability of satisfying the need for such a bulletin board.
Broker/Dealer and IA/IC participants felt that it would be beneficial to make copies of significant examination reports and examination records (e.g., deficiency letters, responses) available on-line to examiners. Management indicated that a computer database file and search application known as the Zy INDEX is currently being implemented in parts of the examination function. This technology will enable examiners to research examination information through on-line access.
Workshop participants expressed a desire to receive certain informational documents. For example, several IA/IC participants mentioned that they did not receive copies of Commission's News Clips. In addition, some B/D participants mentioned that it would be useful to receive copies of articles of interest from publications, such as Investor Business Daily, Wall Street Journal, Morningstar, Blue List, and Investment Lawyer.
Recommendation T
The Office of Compliance Inspections and Examinations should distribute statements of its mission, goals and objectives to all examiners. Any changes to these should be formally reported to the staff.
The Office of Compliance Inspections and Examinations management informed us that mission statements had been widely distributed and that extensive efforts had been made to inform regional office staff of developments in OCIE.
Recommendation U
The Office of Compliance Inspections and Examinations, in coordination with OIT, should develop a detailed action plan for establishing its presence on the Commission's "Intranet". The plan should identify the objectives and information to be kept, as well as specific implementation milestones and timelines. OCIE should consider participant concerns and recommendations in developing the plan.
Recommendation V
The Office of Compliance Inspections and Examinations should evaluate the Zy INDEX after 12 months to determine if it is achieving its intended purpose, modifications can be made to enhance its capabilities, and this technology can be applied to other facets of the examination function.
Recommendation W
The Office of Compliance Inspections and Examinations should ask the Office of Public Affairs to survey the field offices to determine the need for additional copies of the Commission's News Clips. Similarly, the Commission library should determine the need for the various industry publications (identified above) at each field location.
PERFORMANCE MEASURES
Broker/Dealer and IA/IC participants agreed that they generally understood what was expected of them and felt that overall performance standards for examiners were appropriate and related to their jobs. Participants also felt that there should be a correlation between ratings of staff and the respective office performance evaluation by OCIE.
Participants in all three examination areas identified a lack of quality measures for examinations as a material obstacle in evaluating performance. They felt that both the quality and quantity of examinations should be consistently weighed in assessing program goals. In the absence of quality measures, examiners felt that there may be excessive focus on quantity. The most frequently mentioned broad measures of examination quality related to findings, workpapers, and examination reports.
Broker/Dealer participants mentioned that measures related to B/D managers' responsibility to develop and train staff should be strengthened to encourage and improve the effectiveness of training provided to examiners.
Recommendation X
The Office of Compliance Inspections and Examinations should provide examiners additional guidance on how examination quality will be measured. The Office of Administrative and Personnel Management (OAPM), and regional and district offices should consider including such measures in staff performance standards.
The Office of Compliance Inspections and Examinations management informed us that a memorandum is issued to all regional and district directors prior to fiscal year performance evaluations. These memorandum solicited activities and accomplishments from each region, as well as provided guidance on how OCIE would evaluate examination performance.
Recommendation Y
The Office of Compliance Inspections and Examinations should consider including managers' responsibility to develop and train staff as part of its evaluation of regional and district office programs.
Recommendation Z
The Regional and District Office management, in coordination with OAPM, should review the performance standards of B/D managers to determine what enhancements, if any, can be made regarding managers' responsibility to develop and train staff.
STAFF RETENTION
Good benefits, interesting work, pleasant environment and overall prestige of working at the Commission were universally seen as advantages to working in the examination function. However, low compensation, compared to the private sector, was seen as the main disadvantage of being an examiner.
High turnover was a serious concern expressed during the workshops. Participants overall rated the achievement of the staff retention objective the lowest of the eight supporting objectives. Management and staff participants from all three examination areas believed that many obstacles impair the examination function's ability to implement the staff retention objective and that it was not being achieved.
Certain aspects of staff retention (e.g., base pay, bonuses) are out of the control of OCIE or Commission management because of government-wide restrictions. Participants, however, had several specific suggestions for increasing staff retention including making the "journeyman" grade of an examiner a GS-14 and recruiting examiners with industry experience at higher grades. OCIE management has been working with OAPM to enhance its ability to use retention pay and bonuses. OAPM has worked with the Office of Personnel Management, Office of Management and Budget, and the Congress to obtain exemptions from existing regulations.
Recommendation AA
The Office of Administrative and Personnel Management, in consultation with OCIE, should consider the feasibility of the participants' proposals (making the "journeyman" grade of an examiner a GS-14 and recruiting examiners with industry experience at higher grades).
[APPENDIX A]
CONTROL SELF ASSESSMENT METHODOLOGY
The Control Self Assessment (CSA) methodology used consists of four primary tasks: identify management objectives, convene workshops to discuss and rate each objective, evaluate the workshop data, and prepare an audit report.
Management Objectives
At the beginning of the audit, the OIG audit staff worked closely with management to develop concise objectives for the examination function. Examples of possible objectives were initially discussed in several joint meetings of management and auditors. The objectives selected for evaluation were those that management thought were both important and for which evaluation data would be useful.
The primary examination function objective developed by management was:
Conduct Effective Examinations and Inspections of Regulated Entities to Protect Investors
Eight supporting objectives, reflecting the activities that make achievement of the primary objective more likely, were also developed by management. They were:
TARGETING EXAMINATIONS / INSPECTIONS - Select regulated entities for review in order to achieve strategic targeting goals (e.g., risk, routine cycle, oversight).
EXAMINATION / INSPECTION EFFECTIVENESS - Ensure that examinations and inspections are effective in uncovering significant non-compliance with securities laws, or weaknesses in oversight.
EXAMINATION / INSPECTION FOLLOW-UP - Ensure that regulated entities appropriately address examination/inspection findings in a timely manner.
EXAMINATION / INSPECTION EFFICIENCY - Ensure that resources are efficiently utilized and management controls are effectively implemented.
STAFF TRAINING - Ensure that staff have sufficient training to perform effectively.
COMMUNICATIONS - Keep staff informed of industry trends, and Commission policies and positions.
PERFORMANCE MEASURES - Measure and evaluate individual performance in conformity with program objectives and goals.
STAFF RETENTION - Promote a work environment and career track that fosters the retention of qualified employees.
The Workshops
Prior to the workshops, each participant received a handbook which described what was expected and encouraged them to think about the management objectives in advance. Each workshop began with a 45-60 minute presentation on the work of the OIG and took from 4 to 6 hours to complete.
Two flip charts were used to summarize and record the 1) strengths and 2) weaknesses identified by participants for each objective. Participant recommendations were also solicited and recorded. Prior to the workshops, the auditors developed sets of questions that related to each objective. The questions gave structure to the discussions and facilitated the conduct of the workshops.
At the end of the discussion on each objective, participants were asked to use 4x6 inch cards to anonymously rate how well the management objective was achieved using a scale of 1 to 7. [Appendix A contains the rating criteria used by the participants.] The cards were passed down to one of the participants who read them out loud. Each anonymous rating was recorded on the flipchart. The average was then calculated by OIG staff using an Excel spreadsheet. This rating procedure was then repeated to rate how important (or desirable) the participants thought the objective was. Two ratings (i.e., actual and desired) were collected for each objective.
During each workshop, the auditors typed the summary successes, obstacles, and recommendations from the flipcharts to a formatted word-processing document (one page per objective). Copies of the comment summaries were then mailed to each of the participants who were then asked to review the comments and verify their accuracy.
Data Evaluation
Quantitative Analysis
The anonymous assessment ratings of the workshop participants were keyed into an Excel spreadsheet for quantitative comparison and graphical presentation. Several comparisons (e.g., how well ratings aligned) were performed. These included analysis of actual versus desired performance and opportunities for improvement.
Qualitative Analysis
All of the successes, obstacles, and recommendations ("comments") from the workshop participants were recorded on worksheets and subsequently verified by the participants.
Based on reading through all comments, one hundred resulting issue areas were identified for the three examination areas by the audit staff. Each success, obstacle, and recommendation was then coded and electronically transferred to a "resulting issue worksheet." This brought all comments about a particular issue together in one document and facilitated discussion and evaluation of the issues. Each comment retained a cross-reference code to its original workshop so that the context of the comment could be obtained, if required.
The resulting issue area worksheets for each of the three examination areas were reviewed by both management and the auditors. Several meetings were held to discuss the issues and identify what actions could be taken to enhance the examination function. These analyses were the basis for this audit report.
Although some issues raised were outside of the scope of the audit, management indicated that they found the information extremely useful and had taken corrective actions in many instances. This information is also being used in the OIG's risk assessments, used to select future audit topics.
Audit Report
Although the OIG drafted the audit report, discussions with management and their written comments were used to prepare it. This departure from the traditional comment process provided input at an earlier stage in the writing process. It also gave management a better understanding of the audit results, since they also analyzed the resulting issue worksheets.
Participant ideas, expressed during the discussions preceding the preparation of the audit report, exceeded the expectations of the OIG. We believe that the content of the final report is superior to what it would have been if only comments on the report itself were received from management. We estimate that the process took about the same amount of time as the traditional comment approach.
[APPENDIX B]
RATING SCALES
Actual Effectiveness
7 The Commission is successfully implementing the supporting objective. Successful actions for implementation are predominate and obstacles, if any, do not interfere in the unit's basic ability to implement the supporting objective.
6 Midway between 5 and 7.
5 The Commission is generally implementing the supporting objective. Several successful actions for implementation exist, but some obstacles are impairing the unit's ability to fully implement the supporting objective.
4 Midpoint of the scale.
3 The Commission is generally not implementing the supporting objective. Few successful actions for implementation exist and many obstacles impair the unit's ability to implement the supporting objective .
2 Midway between 1 and 3.
1 The Commission is not implementing the supporting objective in a meaningful manner. Very limited successful actions for implementation exist and obstacles are so prevalent that the unit is significantly impaired from implementing the supporting objective.
Importance
7 The supporting objective is extremely important and the Commission should fully implement it.
6 Midway between 5 and 7.
5 The supporting objective is relatively important and it should be generally implemented at the Commission.
4 Midpoint of the scale.
3 The supporting objective is relatively unimportant and its general implementation should not be a high priority at the Commission.
2 Midway between 1 and 3.
1 The supporting objective is not important and should not be implemented at the Commission.
[APPENDIX C]
SUGGESTIONS FOR BEST PRACTICES DOCUMENT
- Pre-examination planning, including specific examples of best practices should be emphasized. (B/D)
- Examiners should be scheduled as far in advance as possible in order to allow adequate time to conduct pre-examination planning and enable them to better utilize downtime that occurs between examinations. (IA/IC)
- Branch chiefs should go out on examinations in order to assist in the training of new examiners, and observe first-hand the problems encountered by examination teams. (IA/IC)
- Examiners should interview a variety of a firm's staff when on an examination, including directors, operations personnel, and head traders. (IA/IC, SRO)
- Regional managers should conduct face-to-face meetings with B/D executives to discuss serious problems that do not warrant referral to Enforcement. In addition, regional managers should consider what officials (e.g., Enforcement attorney) should attend. (B/D)
- Regional and district managers should conduct brief follow-up examinations on a sample basis. (B/D, IA/IC)
- Regional and district managers should emphasize the purpose, importance, and cost of workpaper documentation to their respect staff. The purpose of documenting significant work performed should be emphasized so that examiners prepare sufficient workpapers. (B/D, IA/IC, SRO)
- Regional managers should hire interns to assist examiners and serve as a source of future new hires. (B/D)
- Regional and district managers should ensure that the size and collective experience of the examination team are appropriate. (IA/IC)
- Regional and district managers should review examination reports and workpapers in a timely manner to promote an effective review process. (IA/IC)
- Regional and district managers, on an annual basis, should distribute the specific goals and objectives of their jurisdiction to their staff. (B/D and IA/IC)
- Regional and district managers should meet with their staff on a regular basis to exchange views about the program, as well as give briefings on specific inspection results. These meetings should consist of an exchange of information. (B/D, IA/IC and SRO)
- Regional and district managers should give examiners performance feedback at the conclusion of each examination. (B/D and IA/IC)
- Regional and district managers should circulate copies of the Commission's News Clips to the examination staff in a timely manner. (IA/IC)
- Regional and district managers should carefully consider if a surprise examination is appropriate. In addition, to minimize examiner downtime during surprise examinations, examiners should prioritize information requests while on-site and schedule opening interviews after document requests have been made to the registrant. (B/D and IA/IC)
Endnotes
1 This training effort occurred subsequent to the ratings and, thus, was not reflected in the views of the workshop participants.
2 In the months before the effective date of the Investment Adviser Supervision Coordination Act, the Commission examined a large number of small financial planners, thus producing an unusually large number of examinations. During a normal year the number of examinations performed would be at the lower end of this range.
3 This rating does not reflect a significant training effort organized by OCIE subsequent to our workshops.
4 This training effort occurred subsequent to the ratings and, thus, was not reflected in the views of the workshop participants.
5 This rating does not reflect a significant training effort organized by OCIE subsequent to our workshops.