This document is an HTML formatted version of a printed document. The printed document may contain agency comments, charts, photographs, appendices, footnotes and page numbers which may not be reproduced in this electronic version. If you require a printed version of this document contact the United States Securities and Exchange Commission, Office of Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202) 942-4460.
Enhancing Excellence - Integrity Program
Audit Report No. 250
Actual Effectiveness Rating
CONDUCT OF MANAGERS
SENSITIVITY TO UNINTENDED CONSEQUENCES
LINK INTEGRITY TO PERSONNEL DECISIONS
FAIR NOTICE OF CONDUCT PARAMETERS
INTERNAL MANAGEMENT CONTROLS
The composite ratings of actual achievement by the participating staff and managers indicate that all supporting objectives are generally being implemented, although some obstacles are impairing full implementation. We believe that, taken as a whole, the Commission is achieving its primary objective to promote high individual and agency integrity.
An important theme, expressed overwhelmingly by the participants, is the personal commitment of Commission employees to maintain the Commissionís high standards of integrity. Most of the participants also indicated that they felt a strong sense of an ethical tradition at the Commission and that the staff live up to the Commissionís integrity expectations. These factors are crucial to an effective integrity program and their existence is a powerful argument that managementís objectives are being achieved.
The participants in the workshops also expressed a desire for better communication from management; more frequent ethics training; readily accessible and understandable material concerning ethics and conduct issues; selective, responsive, and well-trained ethics advisors; and enhanced accountability for misconduct.
Although managers consistently rated the achievement of objectives higher than did the staff participants, the ratings of the two groups were reasonably aligned. The staff ratings of the desired effectiveness (i.e., relative importance) of the supporting objectives were generally a little higher than the ratings of the manager workshops, and, generally, the relative trends of their ratings were directly correlated. There was also little relative difference in alignment between the regional staff and headquarters employees. Similarly, support staff and other staff rated the objectives comparably.
Conduct of Managers
Participants rated the conduct of managers as the most important of the nine supporting objectives. The bulk of the observations of managersí behavior cited were positive (e.g., a manager who stressed that government lawyers have a higher standard of ethics than other lawyers, junior staff learn from senior staff what is expected, managers are expected to behave ethically and violations would only be noticed by staff). These observations support the Commissionís goal that managers act as role models for their staff.
Workshop participants stated that many managers go out of their way to provide meaningful and appropriate assistance to investors. Also impressive was the overwhelming indication that, in spite of pressure to increase output, managers accurately reported workload and other relevant information. Overall, the bulk of the participants reported that integrity is a high priority at, and an integral value of, the Commission.
The participants generally believed that managers are strongly committed to the highest ethical standards and exhibit a high level of integrity. They also found that managers generally make appropriate, ethical decisions on a day-to-day basis.
The participants stressed the importance of communicating integrity goals, as well as policies and their rationale, to the staff and recommended that managers put forth their best efforts to do so. As in many large organizations, some participants inferred that some management decisions may have been based on favoritism or other non-meritorious reasons. Lack of communication was advanced by some as an explanation of why some personnel decisions (e.g., appraisals, promotions) may have appeared arbitrary. The need for a procedure to allow staff to voice disagreement with management was noted as an opportunity for improvement with relative frequency. There is also a need for management to improve communication with the staff and explain why one course of action was chosen over another, when appropriate.
Several support staff thought that managers took some standards (e.g., financial conflict of interest) more seriously than administrative standards (e.g., personnel classification standards). There were also observations of managers who were selected as managers for reasons other than management skill (e.g., their legal skills) and who failed to deal effectively with their supervisory responsibilities.
The Office of Administrative and Personnel Management (OAPM) should modify management courses to instruct managers how to better communicate policies and the rationale for decisions to their subordinates, so that staff will be less likely to misconstrue their motives. Basic management techniques (e.g., how to motivate and relate to diverse staff) should also be included. OAPM should also consider ways in which sound communication can be made a performance standard for managers. Required management training courses should be expanded, if necessary.
The Office of General Counsel, in conjunction with OAPM, should implement and publicize a two part procedure that allows staff to raise professional responsibility questions and other concerns to managementís attention without reprisal. First, staff should be strongly encouraged to direct their concerns or recommendations to their supervisors for discussion and resolution. Secondly, there should be a legitimate option for staff (before or after contacting their supervisors) to raise the issue directly to the Office of General Counsel (or Office of Chief Accountant if appropriate) without reprisal. The procedure should also recognize the existing laws and regulations that require staff to report instances of fraud, waste, or corruption (again without fear of reprisal) to management or the Office of Inspector General.
The OAPM should publicize that the Suggestion Program will accept recommendations for general management improvements.
OAPM should urge managers to hold meetings, at least quarterly, with their staff to explain the rationale for major decisions and changes in policies and priorities.
OAPM should work with selecting officials to place greater emphasis in rating plans on managerial skills (including the ability to communicate, delegate, and review work) and leadership. Management and leadership skills are more critical to managerial job success than technical and legal/accounting skills.
Overall, participants almost universally reported that the organizational climate of the Commission was very supportive of integrity. There was a strong perception among most participants that the staff live up to the Commissionís ethical expectations. They also were generally impressed by the message from the top (e.g., speeches to the industry advancing ethics, memoranda to the staff on integrity topics).
Participants generally agreed that Commission staff readily recuse themselves from matters in which they have a personal connection. However, some were critical about possible overuse of the recusal process, recusal decisions made regarding specific matters, and inefficiencies caused by recusals from a whole matter (in which significant staff time was invested) based on a single connection to a single party associated with the matter.
Several participants suggested that prestigious firms and former employees tend to receive faster service than small firms and people who have no previous association with the Commission. However, the participants raised several explanations, other than favoritism, to explain the phenomenon (e.g., better understanding of Commission rules and procedures allows some to profit from this knowledge). Significantly, there was general agreement that the ultimate outcome of a matter is not substantially changed by the identity of the individual or firm, rather that occasional faster service received was the only benefit.
The participants strongly expressed their impression that there were double standards relating to time and attendance, dress codes, and travel policies. They also perceived that, based on their experience, the Commission had unwritten policies for promotions, bonuses, and performance criteria. There was limited discussion (and no conclusion) of whether decentralized administration of these policies contributes to the appearance of arbitrariness to the staff affected.
Only a relatively small number of participants indicated that they had witnessed what they felt was "inappropriate" interference in their work by superiors. Many of these staff agreed that management did not explain their actions and that this lack of communication could have been a factor in their perceptions of the appropriateness of the intervention. One participant complained that Commission settlements do not resolve charges against the staff raised by the defense at the same time that the securities charges are settled.
The Office of General Counsel, in consultation with appropriate divisions and offices, should widely distribute an analysis of policies related to matters of concern expressed by workshop participants. Among the topics that should be included are:
Why charges against Enforcement staff are settled separately from the securities violations,
The role of attorneys at the Commission (i.e., are they advocates for managementís positions or furnishing independent legal opinions)
Why there is no "representation section" in Action Memoranda to indicate who is representing whom, and
Considerations relating to recusal.
In management training sessions, OAPM should address complaints of inconsistent interpretation and enforcement of administrative policies and reported poor communication of policy rationale by managers.
The Division of Enforcement should formalize the program in which Commissioners and senior Commission managers periodically visit regional and district offices to reinforce integrity policies. In addition, integrity issues should be further emphasized at Enforcement training functions.
Sensitivity to Unintended Consequences
The intent of this objective was to promote staff objectivity in their official interactions with the private sector to prevent unfair impact on persons outside the Commission.
Participants generally seemed well aware of the impact that their actions could have on the public. However, they also acknowledged that some consequences of their actions are unavoidable. For example, the Commission has a general policy of giving no comment about on-going investigations. However, the policy has the unintended consequence of allowing some investors, who call the Commission to check on the advisability of investing in a particular venture, to invest in the venture when the staff has evidence that it is fraudulent.
Many workshop participants were able to cite overt actions taken by the staff to avoid unintended consequences where mitigating actions were possible. For example, there is a new standard paragraph in disclosure review comment letters that gives smaller firms a greater access to supervisors (that was previously enjoyed only by larger firms). Also, the staff can notify subjects named in the Action Memorandum that an investigation is being closed with no action. Moreover the participants were able to identify many controls that were designed to prevent unintended consequences, including ethics rules and procedures, the action memorandum process, the comment and review process, a dual review process for certain matters, a strict press policy, speech pre-clearance procedures, and regular and close supervision of the staff. Several participants indicated that it was sometimes difficult to know where to refer members of the public who had questions or complaints.
Many Commission administrative policies were cited by participants as having an unintended impact on Commission staff. For example, several participants indicated that the number of cases brought was the only measure used for performance appraisals and promotion decisions. (Later discussions with management indicated that this is not policy.) The participants pointed to their perception of the policy as to why some staff are reluctant to take on cases that take a long time (e.g., that might be complex) and look for easy sanctions that will settle. Lack of information about policies and their rationale were cited as a major part of the unintended impact of administrative policies on the staff.
Some participants felt that some Commission policies hindered their ability to get a better understanding of industry participants in order to become more sensitive to unintended consequences. For example, investigators are not allowed on examinations, where they would be able to observe industry practices. The participants felt that this policy exists because regulated entities would be concerned at the presence of an enforcement attorney, and would bring its lawyers into the examination.
OAPM, in consultation with the Office of Investor Education and Assistance, should include the phone numbers of the Commissionís Office of Investor Education and Assistance, other consumer affairs offices, state securities commissions, and self-regulatory organizations prominently in the staff phone book (or on a sticker for the phone instrument) so staff can quickly refer calls from investors. There should also be a short list of rules on where to refer certain kinds of calls (e.g., inquiries about auditing rules) or what can be said (e.g., prohibition on referring investors to private law firms).
The Office of Public Affairs should disseminate the Commission policy on dealing with press inquiries to the staff. It should also coordinate with the training programs in the Commission to include dealing with the press as a topic, where appropriate.
The Division of Enforcement and Regional Directors should establish an instructional program in which young enforcement attorneys would tour the operations of cooperating broker-dealers and investment companies. The tours should focus on day-to-day operations and issues relevant to current enforcement priorities. Enforcement attorneys should be encouraged to participate in these sessions to get a broader understanding of the securities industry and referrals from OCIE.
The Division of Enforcement should issue a memorandum to all enforcement staff (division and regional) explaining the criteria that are used in judging staff performance. It should also review the performance standards of attorneys in the division and regions to ensure they recognize the quality of investigative work, and not just on the result of the investigation, as many staff believe.
Link Integrity to Personnel Decisions
Most participants felt that, although there is no specific performance standard for integrity, it is inherent in existing standards and is informally factored into performance appraisals. Although there were discussions of adding an explicit integrity standard, many felt that it would not be feasible. One participant noted that an existing standard that requires staff to be honest and objective has been successfully used in court to characterize staff testimony.
Many participants reported that they did not know of any instances in which unethical staff were promoted or received awards or bonuses. However, several other participants knew or heard of someone who had been promoted despite unethical behavior. For example, several of them had heard that managers involved in a sexual harassment suit of several years ago, that went against the Commission, were promoted or given bonuses.
Also, a small number of participants felt they had been retaliated against (or threatened with retaliation) for acting, in their opinion, with integrity. For example, one participant reported being threatened with a delayed promotion for not backdating performance standards.
In consultation with the OAPM, OGC, EEO, and OIG, the Executive Director should periodically (e.g., annually) publish an analysis of disciplinary penalties imposed for staff misconduct for Commission staff. The analysis should include a rough outline of the charges and associated penalties, but carefully avoid disclosing the identity of the subject or reveal identifying details.
Fair Notice of Conduct Parameters
With almost no exceptions, all participants had received the Commissionís ethics manual. However, it was noted that some managers hold their staff to stricter standards of ethical conduct than those documented in the manual. The ensuing discussions produced numerous constructive suggestions for improving the utility of the manual to the staff. Among these suggestions were: annotate the manual with examples, put it on the Local Area Network (LAN) with a key word search capability, develop a section on "Frequently Asked Questions", include a table of penalties, and incorporate integrity memoranda on administrative topics (e.g., sexual harassment, frequent flyer miles).
According to the Office of Ethics Counsel, it established an ethics bulletin board on the Commission's LAN during the audit. The ethics bulletin board is expected to provide an efficient and effective new mechanism for the Office to disseminate ethics related guidance and information to the staff and management. The bulletin board will also provide a means to communicate the Commission's integrity goals to the staff. In addition, the Office of Ethics Counsel will consider requests from Division Directors,
Office Heads, and Ethics Liaison Officers to utilize the bulletin board to transmit ethics related information tailored to the staff in their office or division.
Most participants were not familiar with the Personnel Operating Policies and Procedures (POPPS) manual. The POPPS manual is a compilation of some of the administrative policies of the Commission. Several participants were told by their supervisors that the manual was for management only and not available to staff. Although restrictions on the distribution of the POPPS manual are contrary to managementís stated objectives, most participants felt that they did have fair notice of acceptable and prohibited behaviors.
With the exception of staff from the support offices, almost none of the participants were aware of the existence of the Table of Penalties or its contents 1/. Moreover, because of the Commissionís attempts to protect the privacy of staff involved in disciplinary actions, almost none of the participants knew of penalties associated with specific types of misconduct. Together, these findings indicate that staff do not have "information on the consequences of non-compliance," which is part of the supporting objective.
The Office of Ethics Counsel should notify staff and management of the ethics bulletin board and its goals. Division and office management should be contacted to solicit their involvement in using the bulletin board to transmit ethics information tailored to their staff.
OAPM should notify managers and staff that full access to the POPPS manual by all staff members is permitted.
The Office of Executive Director, in conjunction with OAPM and the Comptroller, should develop and implement an effective means of timely answering administrative questions from the staff and managers. Workshop participants suggested several approaches to the problem including a neutral person who would act as a staff representative to answer questions about administrative rules, establishing an administrative duty officer of the day, or an administrative hotline that staff could call for interpretations of administrative matters. Another approach may be to use an E-Mail address to which staff could write administrative questions or seek interpretations of rules; an assigned person would forward each question daily to the appropriate official for a timely response.
The Office of Ethics Counsel should consider the revisions to the Ethics Manual suggested by workshop participants discussed above. The Office of Ethics Counsel should also assist divisions and offices to document any ethics standards tailored to their specific organization.
OAPM should implement its plans for an Employee Handbook that will summarize key information about policies, benefits, services, and other information and that will refer staff to other manuals for more specific information, as appropriate.
OAPM should complete its restructuring and improvement of the POPPS manual to simplify its style and make it easier to locate information.
In conjunction with the Office of Ethics Counsel, OAPM should complete its plans to reconsider the orientation program for new employees, with the intention of following up with a more in-depth ethics/integrity presentation within a few months. This effort should be coordinated with the development of the Employee Handbook.
OAPM and the Office of the Comptroller, in conjunction with the Office of Information Technology (OIT), should make the POPPS, Time and Attendance, and Travel manuals available on the Commissionís LAN to ensure staff access. When these manuals are available on the LAN, the staff should be notified.
Several participants complemented the Office of Ethics Counsel and OAPM for their effectiveness and efforts. Other participants reported that staff informally "razz" other staff about minor rule infractions and that this peer pressure helped prevent further violations of the rules. Participants in general indicated that they would report fraud, waste, or corruption to their supervisors or the Office of Inspector General. Although a few participants worried about reprisals for reporting misconduct, most did not express such fears.
Most participants had no knowledge of any Commission disciplinary actions. The bulk of those who knew of disciplinary actions thought that the matters were handled fairly by the Commission. Several of those also reported that they were not informed of whether the matter was closed or how it turned out (even if they were preliminary subjects or witnesses).
Most workshop participants felt that the Commissionís policy of not disclosing summary information about disciplinary actions was dysfunctional. They thought the policy generated rumors, eliminated the deterrent effect of the penalty, generated mistrust among the staff, and resulted in staff believing that misconduct was not punished.
Participants described numerous instances in which misconduct and poor performance were ignored by managers. Several different explanations were advanced. Some workshop participants indicated that in many of these instances a managerís failure to supervise was overlooked by his/her superiors. They also observed that sometimes minor misconduct is allowed to slide and then managers overreact when the situation gets out of hand.
Some participants noted that many staff are resentful of doing the work of others who are not held accountable for poor performance. Ironically, some professional and support participants each felt that the other group was generally treated more leniently than their own group.
OAPM should prepare sample supervisory performance standards for various grade levels and encourage their use throughout the Commission. The sample standards should emphasize that managers are held accountable for their supervisory performance, as well as the performance of their programs.
Staff counseling received the highest composite rating for actual effectiveness of all the supporting objectives evaluated by the participants. The Office of Ethics Counsel generally received high marks from the participants for its work.
Ethics liaisons in the divisions and offices were the primary source of ethics advice for most participants and were generally held in high regard by the participants. However, isolated comments about liaisonsí lack of training and the lack of uniformity of advice from liaisons were expressed.
Most workshop participants were unsure of how confidential their conversations with the ethics liaisons were (e.g., whether liaisons were prohibited from discussing the content of conversations with management, whether an attorney/client privilege attached, whether the liaison was bound to report fraud reported to them) or whether any immunity from future prosecution was provided if the advice of the ethics liaisons was followed.
Although some participants thought that some liaisons were too strict (e.g., staff were advised that they could not accept cookies from the officeís landlord at Christmas), there was general agreement that their advice was followed by Commission staff.
Most participants appeared to be unaware that OAPMís Employee Relations group was available for consultation.
In order to provide more consistency, the Office of Ethics Counsel should provide regular formal training for ethics liaisons and their deputies.
The Office of Ethics Counsel should provide staff with an explanation of the extent to which conversations with liaisons are confidential and the extent to which their advice can be relied upon.
Most of the participants that joined the Commission within the last several years did not recall receiving integrity training at their first day orientation; only that they received the Ethics Manual. The last integrity (ethics) training most of the participants remember receiving was a video on receiving gifts, shown several years ago.
The participants generally thought the video on receiving gifts was effective training. Although most had favorable impressions of articles in the Employee News and memoranda on various integrity related topics, some thought they were unnecessarily wordy. Many also thought the topics should be more closely tailored to the work of the training recipient.
During workshop discussions, the participants had numerous suggestions for improving integrity training. Suggested topics included: staff securities transactions; use of CRST; examples illustrating rules about disclosures to the public, press, and other agencies; recent changes in integrity rules; conduct regulations; exceptions to general rules made by a particular office; disclosure guidance for secretaries (especially on answering phone inquiries); administrative policies such as frequent flyer miles, travel, and time and attendance; how to handle personal disagreements with management decisions; outside employment; how to recognize bribes (e.g., a job offer); and the duty of staff to report possible bribes.
According to the Office of Ethics Counsel, it has conducted a series of ethics training sessions on negotiating for employment and post-employment issues. The training was held in November and December of 1996. The Office of Ethics Counsel is also working with OAPM to improve the current new employee orientation training. To the extent possible, training will be targeted to the particular offices.
Training methods suggested by workshop participants included: video-taped orientation for new employees; administering an ethics test to employees; a video using role-playing to illustrate important integrity issues; each supervisor train his/her own employees on enumerated integrity topics (i.e., cascade training); continue the articles and memoranda on integrity topics; and put periodic (e.g., biweekly) E-Mail messages to all employees on the LAN.
According to the Office of Ethics Counsel, in conjunction with OIT, it is also developing an ethics library site on the LAN. OIT is currently considering using an "Intranet" system, that is similar to the "Internet", but will be accessible to Commission employees only. The Office of Ethics Counsel is preparing the following items to be included on the Intranet: an updated Ethics Handbook, interactive training modules, and a summary of the ethics regulations.
The information located on the Intranet site will be organized by subject matter headings. Under each heading, employees will be able to find related memoranda, statutes, regulations, forms, separate office policies, and training modules. Also, to help employees quickly search through data on the Intranet, search capabilities similar to Lexis and Westlaw are being explored. The Intranet will provide a means for creating a central location where employees may find the various office and division policies, memoranda, and directives on employee conduct.
The Office of Ethics Counsel, in conjunction with OAPM, should develop periodic (e.g., annual) integrity training for Commission staff. The offices should consider incorporating the topics and methods suggested by workshop participants, noted above.
The Office of Ethics Counsel should approach each division and office and urge them to include customized integrity training as a part of their program training sessions. This might also entail the Office of Ethics Counsel providing technical assistance to the division or office in preparing customized integrity training.
OIT and the Office of Ethics Counsel should implement the ethics library on the LAN, as soon as possible.
Internal Management Controls
The actual effectiveness of internal management controls received the second highest rating of all the supporting objectives by the workshop participants. There were no material control weaknesses identified by any participant during the workshops.
At the request of the auditors, workshop participants identified many risks that are inherent to Commission operations. They also cited effective internal management controls for these risks. Several previous significant control weaknesses were reported to have been recently strengthened.
Several controls were identified by some participants as being excessive or duplicative (e.g., certified mail, requests for equipment used to prepare budgets, travel vouchers, 30 day
holding period for securities, requiring transactions to be reported after getting permission from CRST, financial reporting in both May and October).
Participants suggested several ways to increase the effectiveness and efficiency of financial reporting controls. There were also several criticisms of the CRST system used to check on the allowability of proposed staff securities transactions. Since the revised Rule 5 changes would eliminate the CRST system, no recommendations on the system are forthcoming in this report.
CONTROL SELF ASSESSMENT METHODOLOGY
The Control Self Assessment (CSA) methodology used consists of four primary tasks: identify management objectives, convene workshops to discuss and rate each objective, evaluate the workshop data, and prepare an audit report.
At the beginning of the audit, the OIG audit staff worked closely with management to develop concise objectives for the conduct and ethics programs. Examples of possible objectives were initially discussed in several joint meetings of management and auditors.
The objectives selected for evaluation were those that management thought were both important and for which evaluation data would be useful. For example, although financial disclosure is important, the Office of Government Ethics had recently conducted a compliance audit of the matter, so it was not selected. Management revised their objectives further based on experience gained in the pilot workshop, the participants of which were staff from the ethics program and personnelís staff conduct program.
The primary integrity objective developed by management was:
Promote High Individual and Agency Integrity
Nine supporting objectives were also developed that reflect the activities that make achievement of the primary objective more likely.
CONDUCT OF MANAGERS - Ensure that the behavior of executives and managers reflects the SECís integrity values and principles and that they acknowledge their critical role in reinforcing these values with their subordinates.
ORGANIZATIONAL CLIMATE - Foster an organizational climate that promotes high standards of ethical behavior.
SENSITIVITY TO UNINTENDED CONSEQUENCES - Promote staff objectivity in their official interactions with the private sector to prevent unfair impact on persons outside the Commission.
LINK INTEGRITY TO PERSONNEL DECISIONS - Ensure that managers consider employeesí ethical behavior when deciding on their performance ratings, awards,
promotions, selection to supervisory positions, or other personnel actions to reward behavior that furthers SEC integrity.
FAIR NOTICE OF CONDUCT PARAMETERS - Provide staff with fair notice of the parameters of acceptable and prohibited behaviors, along with information on the consequences of non-compliance.
DISCIPLINARY PROCESS - Deal with staff misconduct in a fair, consistent, and timely manner.
STAFF COUNSELING - Provide employees with an opportunity to obtain authoritative answers to ethics and integrity questions in order to enable them to make ethical decisions.
INTEGRITY TRAINING - Promote staff awareness and commitment to integrity.
INTERNAL MANAGEMENT CONTROLS - Administer effective financial, administrative, and program controls to deter misconduct.
Once the objectives were finalized, sixteen workshops were convened by OIG personnel. The workshops were set up to be roughly representative of staff allocation within the Commission. For example, because about one-third of Commission staff work in the regions, five of the workshops were conducted with regional staff.
For logistical reasons, division and office heads were asked to select staff to participate in the workshops based on criteria provided by the OIG (e.g., a professional working in the regulation program must have worked at the Commission at least one year). They were also asked to select staff to be as representative as possible, after the auditorís selection criteria were met. Each workshop was designed to be homogeneous with respect to management, professional, or support staff so that comparisons could later be made. The workshop participants were also exclusively either headquarters or regional staff or managers.
Prior to the workshops, each participant received a handbook which described what was expected and encouraged them to think about the management objectives in advance. In preparation for the workshops, the auditors developed sets of questions that related to each objective. The questions gave structure to the discussions and facilitated the conduct of the workshops.
Each workshop began with a 30 - 45 minute presentation on the work of the OIG. A 10 minute video was then presented, in which management introduced the workshop and provided background information. Each of the workshops took from 4 to 6 hours to complete.
The strengths and weaknesses identified by participants for each objective was summarized and recorded on flip charts. Participant recommendations were also solicited and recorded.
At the end of the discussion on each objective, participants were asked to anonymously rate how well the management objective was achieved using a scale of 1 to 7. [Appendix B
contains the rating criteria used by the participants.] The 4x6 inch cards, used to anonymously indicate individual ratings, were passed down to one of the participants who read them out loud. Each anonymous rating was recorded on a flipchart. Another participant simultaneously calculated an average rating using a calculator. This rating procedure was then repeated to rate how important (or desirable) the participants thought the objective was. Two ratings (i.e., actual and desired) were collected for each of the objectives.
After each workshop, the auditors typed the summary successes, obstacles, and recommendations from the flipcharts to a formatted word-processing document (one page per objective). Participants were asked to review the worksheets from their workshop and verify its accuracy.
The anonymous assessment ratings of the workshop participants were keyed into Statistical Analysis System (SAS) software for analysis. Comparative results from SAS were transferred to Excel software for graphical analysis and presentation. Several comparisons (e.g., how well ratings aligned) were performed. These included analysis of:
Management versus staff,
Regions versus headquarters,
Clerical versus professional,
Actual versus desired performance, and
Opportunities for improvement.
The detailed graphs from the Excel software were shared and discussed with program management.
All of the successes, obstacles, and recommendations ("comments") from the workshop participants were recorded on worksheets and subsequently verified by the participants. Seventy-five "resulting issues" were identified by the audit staff by reading through all of the comments. Each success, obstacle, and recommendation was then coded and electronically transferred to a "resulting issue worksheet." This brought all comments about a particular issue together in one document and facilitated discussion and evaluation of the issues. Each comment retained a cross-reference code to its original workshop so that the context of the comment could be obtained, if required.
The seventy-five resulting issue worksheets were reviewed by both management and the auditors. Several meetings were held to discuss the issues and identify what actions could be taken to enhance the integrity program. These analyses were the basis for this audit report.
Although the scope of the audit was Commission-wide, the workshops also provided information with respect to individual organizational units. The original worksheets from the individual workshops (which did not associate individuals with their comments or ratings) were shared with the management of those units. Although some of the issues raised were outside of the scope of the audit, management indicated that they found the information extremely useful and had taken corrective actions in many instances. This information is also being used in the OIGís risk assessments, used to select future audit topics.
The OIG prepared the audit report. The "resulting issue worksheets" were given to management to review. Their written comments and the results of meetings to discuss the worksheets preceded the preparation of the audit report. As is the Officeís normal policy, several drafts of the audit report were circulated for comment before the report was finalized.
This expanded comment process provided input at an earlier stage in the writing process. It also gave management a better understanding of the audit results, since they were also able to analyze the resulting issue worksheets.
EFFECTIVENESS RATING SCALES
7 The Commission is successfully implementing the supporting objective. Successful actions for implementation predominate and obstacles, if any, do not interfere in the unit's basic ability to implement the supporting objective.
6 . . . . . .
5 The Commission is generally implementing the supporting objective. Several successful actions for implementation exist, but some obstacles are impairing the unit's ability to fully implement the supporting objective.
4 . . . . . .
3 The Commission is generally not implementing the supporting objective. Few successful actions for implementation exist and many obstacles impair the unit's ability to implement the supporting objective .
2 . . . . .
1 The Commission is not implementing the supporting objective in a meaningful manner. Very limited successful actions for implementation exist and obstacles are so prevalent that the unit is significantly impaired from implementing the supporting objective.
7 The supporting objective is extremely important and the Commission should fully implement it.
6 . . . . .
5 The supporting objective is relatively important and it should be generally implemented at the Commission.
4 . . . . .
3 The supporting objective is relatively unimportant and its general implementation should not be a high priority at the Commission.
2 . . . . .
1 The supporting objective is not important and should not be implemented at the Commission.
|Home | Previous Page||