This document is an HTML formatted version of a printed document. The printed
document may contain agency comments, charts, photographs, appendices,
footnotes and page numbers which may not be reproduced in this electronic
version. If you require a printed version of this document contact the
United States Securities and Exchange Commission, Office of Inspector General,
Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C. 20549 or call (202)
942-4460.
Comment Letter Process
Audit Report No. 259
February 2, 1998
INDEX
Executive Summary
Scope And Objectives
Audit Methodology
Background
Audit Results
Overall Results
Communicate
Staff Findings
Management Review
Communicate Comments
Resolve Comments
Program
Performance Measures
Staff Performance
Measures
Appendix A - Control Self Assessment Methodology
Management Objectives
The Workshops
Data Evaluation
Quantitative
Analysis
Qualitative
Analysis
Audit Report
Appendix B - Effectiveness Rating Scales
Actual Effectiveness
Importance
Executive Summary
The Securities and Exchange Commission's Office of Inspector General (Office)
evaluated the Division of Corporation Finance's (Division) comment letter
process. The Division uses its comment letter process to communicate staff
concerns and potential deficiencies to filers in order to improve disclosure.
We obtained and analyzed information concerning successes, obstacles,
recommendations, and effectiveness ratings related to the primary objective
and six supporting objectives identified by the Division for the comment
letter process. To obtain this information, we conducted six internal audit
workshops involving approximately seventeen per cent of Division staff.
Separately, we also reviewed feedback from issuers on the process and analyzed
information on comment letters and filing processing to determine conformance
to Division procedures and timeliness goals.
Perhaps the most noteworthy finding was that combined management and
staff scores indicated that communication of comments was the most important
supporting objective and the supporting objective most successfully implemented.
Management and staff also agreed that the Division is composed of, for
the most part, hardworking, conscientious, highly qualified and motivated
staff who are dedicated to improving disclosure to investors. Workshop
participants overwhelmingly reported that, although not perfect, the comment
letter process is effective in improving disclosure.
The participants in the workshops expressed a desire for better communication
of Division filing review policies from management to staff; more consistent
treatment of filing issues; clearer definition of accountant, examiner,
and reviewer responsibilities; and re-evaluation of the filing examination
report format. The Office generally endorses these recommendations and
makes additional recommendations in its report. These recommendations include
updating review guidance as appropriate, designing and implementing formal
training for reviewers, and improving the use of outcome-based program
performance measures.
Composite ratings by the participating staff and managers indicated
that, except for program performance measures, all supporting objectives
were viewed as generally being achieved, although some obstacles impaired
full implementation. Both managers' and staff's scores indicated that the
program performance measure objective was generally not achieved. The staff
and management disagreed about whether the measurement of staff performance
was achieving the Division's objectives. The staff thought the objective
was not being achieved; management rated it as generally being achieved
although some obstacles impaired full implementation. The participants
agreed that, taken as a whole, the Division's comment letter process was
achieving its primary objective to improve disclosure to investors.
The Division gave us information on issuers that were contacted by the
Chairman within the past year for their views on the filing review process.
After reviewing the information we contacted five companies not on the
Division's list for their feedback on the Division's comment letter process.
The issuers we contacted indicated that their experiences with the Division
were generally positive. Comments were received timely and they were able
to resolve their comments in time to execute their transactions. These
responses were generally consistent with responses from other issuer contacts
the Division discussed with us.
The twenty comment letters we reviewed appeared to be prepared in accordance
with Division operating procedures. We also reviewed a list of 1933 Act
registrations filed between October 1, 1996 and June 31, 1997 to determine
the timeliness of filing review comments. Our review indicated that the
Division issued comments within its 30 day timeframe on approximately 88%
of the 1933 Act filings in our sample receiving full reviews and approximately
98% of the 1933 Act filings receiving a monitor for one or more specific
items.
SCOPE AND OBJECTIVES
The primary objective of this audit was to evaluate how well the Division
achieves its objectives with respect to the timeliness and effectiveness
of the comment letter process. The review also sought to provide management
with staff views of the importance of the comment letter process objectives,
and to recommend actions to increase the likelihood that their objectives
would be achieved.
During the audit, management designated the primary objective of the
comment letter process to be:
Improve Disclosure to Investors
Division management also identified six supporting objectives. The supporting
objectives reflect activities that increase the chances of achieving the
primary objective. They were:
Communicate Staff Findings - Document and communicate
to supervisors staff findings from the review of disclosure filings by
preparing an examination report/draft comments.
Management Review - Review the examination report/draft
comments for policy considerations (e.g., materiality, uniformity).
Communicate Comments - Develop and communicate
(written and verbal) comments to the issuers or their representatives.
Resolve Comments - Gain issuer acceptance of comments
through implementing amendments and futures comments.
Program Performance Measures - Measure and evaluate
the performance of the comment letter process accurately.
Staff Performance Measures - Evaluate staff performance
in the comment letter process to improve individual and program performance.
The field work was performed between May and October 1997 in accordance
with generally accepted government auditing standards.
AUDIT METHODOLOGY
A major goal of the audit was to maximize the value of the audit to management,
while maintaining auditor independence. A version of a private sector,
internal audit methodology (Control Self-Assessment or CSA) was adapted
for this purpose. The Institute of Internal Auditors has promoted the concept
internationally for the last several years with outstanding results reported.
The premises for using this approach are common to many applications
that use a team approach to decision making (e.g., teams can process more
and more diverse information than an individual, employees are closer to
the information needed than anyone else, staff can make a good system fail
or vice versa, etc.). The CSA methodology used in the audit consists of
four primary tasks: identify management objectives, convene workshops to
discuss and rate each objective, evaluate the workshop data, and prepare
an audit report. These audit steps are described in further detail in Appendix
A.
We collected data primarily through six workshops (fifty-six staff or
approximately 17% of Division personnel participated in the workshops).
The workshop participants discussed and anonymously rated each supporting
objective. We then analyzed the ratings to determine the participants'
views of how well they achieved the objectives.
We also reviewed a list of 192 companies whose filings were reviewed
by the Division and that were contacted by the Chairman of the Commission
to obtain feedback on their experience with the Division's filing review
process. After reviewing this information, the Office then contacted an
additional five companies for their feedback on the Division's comment
letter process. We also reviewed reports of registrations filed between
October 1, 1996 and June 31, 1997 (1,228 registrations receiving full reviews
and 567 registrations receiving a monitor for one or more specific items)
to determine the timeliness of the initial filing review comments. In addition,
we reviewed twenty comment letters to determine if they were prepared in
accordance with Division operating procedures.
BACKGROUND
The Division of Corporation Finance reviews filings submitted by issuers
intending to offer securities in the public markets. The filings are received
by the Commission, sent to the Division, and then screened for review.
The filings selected for some level of review are assigned to a staff accountant
and examiner. The staff accountant performs the initial review of the financial
statements in the filing for compliance with Generally Accepted Accounting
Principles and other accounting rules and regulations. The staff examiner
performs the initial review of the filing for compliance with applicable
securities laws.
During the filing review, the accountant and examiner maintain contact
with the issuer for clarifications and additional information relating
to the disclosures in the filing. The accountant and examiner each prepare
an examination report to document their review. Also, they list any deficiencies
in the filing's disclosures and propose comments relating to those deficiencies.
After completing the review, designated senior staff (reviewers) look
over the filing and proposed comments. As a result of their review, they
edit and add comments as appropriate. The comments are then faxed to the
issuer.
Comments may consist of requests for supplemental information, amendments
to filings, or improvements in future filings (futures comments). Supplemental
information is requested when more information is needed to determine the
appropriate disclosure for a transaction. Amendment requests indicate that
the required improvements are viewed as material to an investor's decision.
Staff requests futures comments when the improvements are not immediately
consequential, but will improve disclosure.
The Division works with the issuer to resolve the comments. Initially,
the issuer communicates with the staff accountant and examiner. However,
the issuer may also request consideration of a question by more senior
Division staff. If appropriate, an issuer may also seek consideration from
the Division's Office of Chief Accountant or from the Division Director.
AUDIT RESULTS
OVERALL RESULTS
Workshop participants anonymously rated how well the Division achieved
each of its six supporting objectives. The rating scale used by the participants
ranged from 7 (full implementation) to 1 (not being implemented in a meaningful
manner). Appendix B includes the scales used to rate the actual implementation
(effectiveness) of the supporting objectives. The composite ratings for
how well the participants felt that the Division actually achieved its
six supporting objectives were as follows:
Communicate Staff Findings - 5.2
Management Review - 4.8
Communicate Comments - 5.7
Resolve Comments - 5.3
Program Performance Measures - 3.6
Staff Performance Measures - 4.1
An important theme, expressed overwhelmingly by the participants, was
the personal commitment of Division employees to maintain high standards
of quality in filing reviews. Most of the participants also indicated that
they felt a strong sense of responsibility to the investing public and
a desire to live up to the Division's, as well as the Commission's, expectations.
These factors are crucial to an effective program.
The composite ratings above of actual achievement by the participating
managers and staff indicated that, except for program performance measures,
the supporting objectives were generally being achieved, although some
obstacles were impairing full implementation. Both management and staff
gave low ratings to the achievement of the program performance measures
objective (management - 3.90, staff - 3.32). Workshop discussions indicated
that this supporting objective received low scores from participants because
of concern with the Division's use of number of filings reviewed as its
primary measure of program performance.
Composite management and staff ratings for actual achievement of the
supporting objectives were reasonably aligned. Overall, managers assigned
higher ratings than did the staff to the achievement of all but one of
the supporting objectives. Despite the differences in the ratings, the
participants believed that, taken as a whole, the Division was achieving
its primary objective to improve disclosure to investors.
Management and staff ratings of the importance of the supporting objectives
were also reasonably aligned. Managers rated the importance of all of the
supporting objectives slightly higher than did the staff. Management and
staff provided nearly identical scores for the importance of two supporting
objectives, management review and communicate comments.
For at least the past year, the Division received feedback indirectly
from companies on their filing review experience. The Division periodically
provided the Chairman of the SEC a list of companies with recently effective
registrations. In turn, the Chairman contacted some of the companies for
feedback on their experience with the Division's filing review process.
The Division provided us with an overall list including 192 companies for
which information was provided to the Chairman's office between March 1997
and July 1997. Only three of the companies on the list raised issues concerning
the Division's filing review process that resulted in a written response
from the Division to the Chairman's office. In all instances, the Division
confirmed that the appropriate actions had been taken during the filing
reviews.
After reviewing this information, the Office contacted an additional
five companies, not on the Division's list, for their feedback on the Division's
comment letter process. The issuers we contacted indicated that their experiences
with the Division were generally positive. Comments were received timely
and the issuers were able to resolve their comments in time to execute
their transactions. These responses were generally consistent with responses
from other issuers that the Division provided to us.
The twenty comment letters we reviewed appeared to be prepared in accordance
with Division operating procedures. We also reviewed reports of 1933 Act
registrations filed between October 1, 1996 and June 31, 1997 (1,228 registrations
receiving full reviews and 567 registrations receiving a monitor for one
or more specific items) to determine the timeliness of the initial filing
review comments. Our review results indicated that the Division issued
comments within its 30 day timeframe on approximately 88% of the registrations
in our sample receiving full reviews and approximately 98% of the registrations
receiving a monitor for one or more specific items.
COMMUNICATE STAFF FINDINGS
Staff document and communicate findings from the review of disclosure filings
to supervisors by means of an examination report and draft comments. Ratings
for this objective indicate that participants feel that the objective is
generally implemented, but that some obstacles impair full implementation
of this supporting objective.
After reviewing a filing, staff prepare an examination report summarizing
the transaction and identifying any material disclosure issues. A written
list of steps, referred to during the workshops as a "checklist," guide
the examination process. The checklist is a response to a prior General
Accounting Office recommendation that the Division document its review
of filings.
The checklist is generally the same for all the industry groups. However,
in certain industries, such as real estate, banking, and insurance, the
checklist is augmented by specialized requirements for that industry called
industry guides. Other industry groups have manuals containing guidance
on issues specific to that industry. Some industries have no specialized
guidance. Participants emphasized that these checklists and guides, while
useful, are not a substitute for the ability to understand the transaction
in the filing. Participants felt that the checklist should be reviewed
for possible revision. Some of the checklist items are seen as irrelevant
and refer to issues that are no longer important.
Participants were asked to describe risks in the examination report/draft
comment process. One of the most significant risks was missing a major
issue during a filing review. However, review of filings and comments by
reviewers mitigate this risk. Another risk identified was associated with
supplemental materials requested from issuers in response to comments.
This supplemental material is sometimes proprietary information that should
be protected from public disclosure. However, issuers often do not request
this protection. Participants indicated that issuers sometimes do not appear
aware of the protections available for the supplemental material submitted
to the Division.
The Division has training for new staff examiners and accountants. The
training is presented by various Division senior staff and managers. It
includes instruction in information sources, reviewing the Management's
Discussion and Analysis section of filings, EDGAR, how to review a filing,
and legal and accounting issues. Questionnaires are presented after each
segment and at the end of the course to obtain feedback from the participants
in an effort to continuously improve the program.
Participants were generally pleased with the training they received.
They generally rated training on new releases as good. They also felt that
the training manual was improved. In general, the mentoring program, where
senior staff accountants and examiners provide guidance to more inexperienced
staff, was well received, although the quality of the individual mentoring
experience depended on the quality of the mentor. However, they also indicated
that the training needed to focus more on the work that the staff actually
performed. For instance, the training concerned with accounting proposals,
while interesting, was not viewed as particularly useful. Also, broader
access to information on issues previously identified in a particular industry
would also be helpful. Participants indicated that a more direct link between
the training and the filings they actually review would make the training
more effective.
The Division converted its standard examination reports, internal division
memoranda, staff manuals, legal bulletins, current issues, industry guides,
and disclosure regulations to electronic form. The electronic documents
were then placed on the Division's file server. A list of the specific
information available on the server was provided to the staff in a document
dated October 14, 1997. Division staff are now able to access the applicable
guidance on-line. Also, updates will be provided on-line.
Recommendation A
The Division should review the current examination report "checklist" and
eliminate any unnecessary or outdated review steps. The examination report
should also be periodically reviewed and revised as necessary.
Recommendation B
The Division should remind issuers of the rules available to protect any
proprietary supplemental information they provide to the Division and the
procedures used to request the special handling.
MANAGEMENT REVIEW
Management reviews the examination report and draft comments for policy
considerations such as materiality and uniformity with prior comments on
similar issues. Generally, this task is delegated to senior accountants
and examiners. Participants indicated that these senior staff, referred
to as "reviewers," are appointed based on their experience and background
in the particular group. Assistant Directors (ADs) review the most complex
filing issues.
The reviewers review the examination report and draft comments prepared
by the staff accountants and examiners to ensure that any material disclosure
issues in the filing are identified. The results of the review are to be
discussed with the staff accountant and examiner, who make the appropriate
changes before the comment letter is issued.
Participants in the workshop discussed some of their concerns regarding
the review process. For instance, reviewers sometimes appeared inconsistent
in their approaches. Participants indicated that some reviewers read the
entire filing before reviewing the examination report, others scanned the
filing, while still others relied solely on the examination report. Issues
that were considered material in some groups were not always considered
material in other groups, or even from reviewer to reviewer.1
No written guidance has been developed for reviewers.
Also, some reviewers are seen as not willing to discuss their review
results with the staff. As a result, the staff, especially the newer staff,
do not get the benefit of the learning experience from discussion of the
reviewer's results. Some participants felt that some reviewers change the
comments on a filing, or insert new comments, without notifying them. These
participants described instances where they were embarrassed when an issuer
contacted them about a comment they knew nothing about, because the reviewer
inserted it without their knowledge. Management indicated that the reviewer
should give the review staff a copy of the comments that were issued. The
staff should then review what was issued and obtain any necessary explanations.
Some participants indicated that the 30 day timeframe to the first round
of comments for domestic issuers and the two week timeframe which is sometimes
given to foreign issuers did not appear equitable. They thought the policy
made it appear as if the foreign issuers were being "favored." Management
indicated that review of foreign filers was facilitated in an effort to
encourage foreign issuers to register their securities with U.S. markets.
Currently, the two-week timeframe is only provided in unusual cases. Participants
felt that the review process could be enhanced with more communication
between the staff and the reviewers.
Recommendation C
The Division should develop a guide for reviewers to use in reviewing examination
reports. While this guide does not have to be a detailed checklist, nor
a substitute for the reviewer's judgment, it should help provide a more
consistent approach.
Recommendation D
The Division should design and implement formal training for reviewers.
Senior and middle managers and support offices such as Chief Counsel and
Chief Accountant should participate in training. This training should result
in a more consistent approach to review as well as improved communication.
COMMUNICATE COMMENTS
Participants rated communication of comments as the most important of the
six supporting objectives. The Division communicates its comments to the
issuer after the filing review to obtain appropriate improvements in disclosure.
Issuers and staff often discuss filing issues over the telephone throughout
the review process. The review staff faxes comments to the issuer, generally
with a follow-up copy by mail. Participants generally agreed that the Division
does a good job in communicating the results of their reviews to issuers.
Staff participants expressed the desire for Internet e-mail access.
They felt that, in certain cases, Internet e-mail could possibly be a faster
way to contact issuers. The Division has already gotten approval for Internet
e-mail access. The applicable policy is dated August 19, 1997.
Several participants suggested separating legal and accounting comments
in the comment letter. They felt that segregating the accounting and legal
comments in the comment letter would make it easier for the issuer to contact
the originator of the comments. In response, management indicated that
accounting and legal comments were, at one time, presented separately in
the comment letters. However, some duplication of comments occurred in
categories that included both accounting and legal issues, such as in the
Management's Discussion and Analysis (MD&A) section of the filing.
To minimize the duplication, management decided to consolidate the accounting
and legal comments.
RESOLVE COMMENTS
Participants rated resolution of comments as the second most important
of the six supporting objectives. Workshop participants agreed that the
current process works well in resolving comments. Issuers who disagree
with the views of the Division staff may contact successively higher levels
in the Division, including the Director. Although issuers do not always
agree with the Division's proposed changes, they generally still make them.
Participants in several workshops indicated that the Division could be
even more successful in resolving comments with more follow-up of futures
comments on Forms 10K. Sometimes the review staff changes or the workload
becomes such that no time is available for follow up on futures comments.
Participants felt that the EDGAR (Electronic Data Gathering, Analysis,
and Retrieval) system should be improved to make it more useful in filing
reviews. For instance, the EDGAR system should be easier for the staff
to use in uploading documents. Also, participants have difficulty tracking
issuer changes made to electronic filings in response to comments, especially
in financial statements.
The Commission is currently seeking a contractor to modernize EDGAR.
Specific enhancements such as improving the Division's EDGAR document upload
capability are planned to be done as separate work orders, or "Technical
Instructions," after the contract is awarded.
Recommendation E
The Division should review the role of futures comments to determine if
there is any adverse impact on disclosure if there is no follow-up, and
take appropriate action.
Recommendation F
The Division should ensure that the new EDGAR system includes improvements
in document uploading capabilities and in identifying and tracking issuer's
changes to filings, including in the financial statements.
PROGRAM PERFORMANCE MEASURES
Program performance measures are used to measure and evaluate the performance
of the review of filings. Participants agreed that the Division uses the
number of filings reviewed as its primary performance measure. Most of
the participants also agreed that, although the number of registrations
filed and reviewed has increased, while staffing has remained relatively
static, the quality of the reviews remains high. Participants also indicated
that the result of their activities can be expressed better than relying
solely on the number of filings reviewed.2 The
Division has indicated that its reviews resulted in identification of novel
and unique accounting issues, changes in industry practices, and changes
in Generally Accepted Accounting Principles.
Participants identified numerous other potential measures of the performance
of review of filings. These potential measures include: the cost of the
filing review function versus the dollar volume of the markets; the cost
of the review as a percent of the filing fee; the dollar amount of changes
in financial statements based on filing reviews; the number of material
comments on filings with unqualified ("clean") audit opinions; the number
of filings with comments implemented; the number of foreign companies that
file with the SEC; and the value of fraudulent offerings stopped.
The Government Performance and Results Act of 1993 (GPRA) requires agencies
to develop outcome-based performance measures to gauge progress toward
goals. An outcome measure assesses the actual results, effects, or impact
of a program activity compared to its intended purpose. The number of filings
reviewed measures the amount of work performed by the review staff, but
does not gauge the effect of the comments on filing disclosures as well
as outcome-based measures.
Recommendation G
The Division should use more outcome-based program performance measures
to gauge the success of the comment letter process.
STAFF PERFORMANCE MEASURES
Supervisors measure staff performance through an evaluation process using
established performance standards. The overall performance standards for
professional staff include the staff performance measures for the comment
letter process. The standards include six general categories. Each general
category contains an average of five sub-elements. Sub-elements relating
to the comment letter process are distributed throughout the six general
categories. The evaluation process based on these standards is intended
to improve individual and staff performance, including performance in the
comment letter process area. One of the general categories include a sub-element
regarding number of filings reviewed. This sub-element requires a specific
average number of "...exams or other reviews per month." The Division also
uses the number of filings reviewed to justify its budget staffing levels.
Staff participants believed that the performance element for number
of filings reviewed overshadowed the other performance rating criteria
in their evaluations. In fact, some participants felt they were penalized
for their lower production numbers resulting from processing larger, more
complex filings than those used in setting the performance standards. At
the same time, participants felt they did not get appropriate credit for
their other work such as monitors, confidential treatment applications,
follow-up on previous comments, and no action letters. Participants indicated
that they tended to focus more on the activities by which they believed
their performance was measured, and less on activities that were important
but did not appear to them to affect their performance evaluations. They
also indicated that the quality of the reviews sometimes suffered in the
push to achieve numerical goals.
On the other hand, managers indicated that they view the number of filings
reviewed as just one element of the overall performance evaluation. However,
adequate review coverage of issuer filings, given limited staff resources,
is essential to the mission of the Division. According to management, the
benefit of improving reviews of a smaller number of filings would not justify
the loss of reviewing a larger number of filings, albeit at only the current
quality level. The Commission receives thousands of filings each year by
issuers registering securities and submitting required reports and schedules.
Participants also gave us their ideas of the characteristics of outstanding
staff. Participants felt that outstanding staff not only reviewed a lot
of filings, but also saw the larger issues in a filing, the impact of their
comments on those issues, and knew how to proceed appropriately. They demonstrated
a high level of interest in the issues, knew when to push an issue and
when not to, were articulate, and knew the technical literature. Participants
indicated that improving these characteristics in the staff as a whole
would also improve the comment letter process.
We asked participants for their opinions on the pluses and minuses of
working for the Commission, specifically the Division. Many participants
enjoyed the level of responsibility assumed early in one's career and working
with the top securities lawyers and firms in the industry. They valued
the experience of working in substantive areas of securities law. Participants
also valued the extent to which they exercised judgment in their work.
In addition, they described a "collegial" atmosphere. All of the participants
felt that their work made a difference and had an impact on the securities
industry.
On the other hand, staff described a number of concerns. Lack of flexibility
in work hours concerned some staff. Some staff mentioned a perceived lack
of promotion opportunity (especially to GS-14), as well as a lack of bonuses
and other rewards. However, despite these concerns, participants indicated
they were proud to work for the Commission, in particular the Division.
Recommendation H
The Division should communicate to the staff management's contention that
it uses the number of filings reviewed as only one staff performance measure.
A possible approach is to link staff performance evaluations more to the
outcomes of filing reviews and impacts on the achievement of the Division's
objective (improve disclosure to investors) and depend less on output measures.
[APPENDIX A]
CONTROL SELF ASSESSMENT METHODOLOGY
The Control Self Assessment (CSA) methodology used consists of four primary
tasks: identify management objectives, convene workshops to discuss and
rate each objective, evaluate the workshop data, and prepare an audit report.
MANAGEMENT OBJECTIVES
At the beginning of the audit, the Office audit staff worked closely with
management to develop concise objectives for the comment letter process.
Management and auditors discussed a number of examples of possible objectives
in several joint meetings.
The objectives selected for evaluation were those that management thought
were both important and for which evaluation data would be useful. Management
revised their objectives further based on experience gained in the pilot
workshop, which consisted of senior Division management.
The primary objective developed by management was:
Improve Disclosure to Investors
Management also developed six supporting objectives, reflecting the activities
that make achievement of the primary objective more likely. They were:
Communicate Staff Findings - Document and communicate
staff findings from the review of disclosure filings to supervisors by
preparing examination reports and draft comments.
Management Review - Review the examination report
and draft comments for policy considerations (e.g., materiality, uniformity).
Communicate Comments - Develop and communicate
(written and verbal) comments to the issuers or their representatives.
Resolve Comments - Gain issuer acceptance of comments
through implementing amendments and futures comments.
Program Performance Measures - Measure and evaluate
the performance of the comment letter process accurately.
Staff Performance Measures - Evaluate staff performance
in the comment letter process to improve individual and program performance.
THE WORKSHOPS
Office personnel convened six workshops after obtaining the objectives.
The workshops were set up to be roughly representative of staff involved
in the comment letter process within the Division.
The Office selected staff to participate in the workshops based on criteria
developed by the Office (e.g., must have been on staff at least six months).
We designed the workshops to be homogeneous with respect to management
or professional staff to facilitate subsequent comparisons.
Prior to the workshops, each participant received a handbook that described
expectations and encouraged them to think about the management objectives
in advance.
Also prior to the workshops, the auditors developed sets of questions
that related to each objective. The questions gave structure to the discussions
and facilitated the conduct of the workshops.
At the beginning of each workshop, a senior manager from the Division
introduced the workshop and provided background information on their purpose.
Each workshop took from 4 to 6 hours to complete.
During each workshop, Office staff summarized participant discussions
regarding successes, obstacles, and recommendations for each objective.
We summarized the comments on flip charts.
After discussing an objective, participants used 4x6 inch cards to anonymously
rate, using a scale of 1 to 7, how well the Division achieved the objective.
[Appendix B contains the rating criteria used by the participants.] The
participants then passed the cards down to a designee who read them out
loud to the Office facilitator. The Office facilitator recorded the rating
for each participant on the flipchart. The facilitator then repeated the
procedure to rate how important (or desirable) the participants thought
the objective was. We collected two sets of ratings (i.e., actual and importance)
for each of the objectives.
After each workshop, the auditors typed the summary successes, obstacles,
and recommendations from the flipcharts to a formatted word-processing
document (one page per objective). We asked participants to review the
worksheets relating to their workshop and verify their accuracy.
DATA EVALUATION
Quantitative Analysis
The Office keyed the anonymous assessment ratings of the workshop participants
into Excel spreadsheets for analysis. We made several comparisons (e.g.,
how well ratings aligned). The comparisons included analysis of management
versus staff, actual achievement versus importance, and opportunities for
improvement.
Qualitative Analysis
Office staff recorded all of the successes, obstacles, and recommendations
("comments") from the workshop participants on worksheets, and distributed
them to the participants, who subsequently verified the worksheets.
Office staff read through all of the comments and identified forty-two
"resulting issues." Each success, obstacle, and recommendation was then
coded and electronically transferred to a "resulting issue worksheet."
This brought all comments about a particular issue together in one document
and facilitated discussion and evaluation of the issues.
Division management and the Office staff reviewed the forty-two resulting
issue worksheets. They met twice to discuss the issues and identify potential
actions to enhance the comment letter process. This audit report is based
on these analyses.
Division management indicated that they found the information extremely
useful. In many instances, they had already recognized the issues and taken
corrective actions. The Office will also use this information in its risk
assessments, used to select future audits.
AUDIT REPORT
The Office prepared the audit report. The "resulting issue" worksheets
were given to management to review. Their comments and the results of meetings
to discuss the worksheets preceded the preparation of the audit report.
This departure from the traditional audit report comment process provided
input at an earlier stage in the writing process. This also gave management
a better understanding of the audit results, since they also analyzed the
resulting issue worksheets.
[APPENDIX B]
EFFECTIVENESS RATING SCALES
ACTUAL EFFECTIVENESS
7 The Commission
is successfully implementing the supporting objective. Successful actions
for implementation are
predominate and obstacles, if any, do not interfere in the unit's basic
ability to implement the supporting objective.
6 Midway between
5 and 7.
5 The Commission
is generally implementing the supporting objective. Several successful
actions for implementation exist, but some obstacles are impairing the
unit's ability to fully implement the supporting objective.
4 Midway between
3 and 5.
3 The Commission
is generally not implementing the supporting objective. Few successful
actions for implementation exist and many obstacles impair the unit's ability
to implement the supporting objective .
2 Midway between
1 and 3.
1 The Commission
is not implementing the supporting objective in a meaningful manner. Very
limited successful actions for implementation exist and obstacles are so
prevalent that the unit is significantly impaired from implementing the
supporting objective.
IMPORTANCE
7 The supporting
objective is extremely important and the Commission should fully implement
it.
6 Midway between
5 and 7.
5 The supporting
objective is relatively important and it should be generally implemented
at the Commission.
4 Midway between
3 and 5.
3 The supporting
objective is relatively unimportant and its general implementation should
not be a high priority at the Commission.
2 Midway between
1 and 3.
1 The supporting
objective is not important and should not be implemented at the Commission.
Endnotes
1 Division is conducting a study simultaneous
with this audit in an effort to increase uniformity.
2 For example, the Division contends that as
a result of staff comments in FY 1996, forty-one companies were required
to restate their income. Of these, 90% resulted in changes in income of
10% or more. These outcomes of staff filing reviews were documented in
"quantifiable benefits" memos. Also, in a 1996 speech, senior Division
management indicated that staff comments resulted in increased issuer attention
in a number of disclosure areas, including accounting for disposal of assets
as discontinued operations, presentation of overly broad geographic segments,
and use of valuation allowances. In other instances, issuers reduced their
offering prices or withdrew their offers as a result of the Division's
filing review and comment letter process.
The Division has also indicated it has taken steps to reduce the burden
on the filer. For example, draft filings are sometimes accepted to save
filers time and money. Also, in August 1995, the Chairman organized the
Task Force on Disclosure Simplification to review and streamline regulations
relating to forms and disclosure requirements. The Task Force's recommendations
have thus far resulted in elimination of forty-six rules and six forms.
_______________________
http://www.sec.gov/about/oig/audit/259fin.htm
