This document is an HTML formatted version of a printed document.
The printed document may contain agency comments, charts, photographs,
appendices, footnotes and page numbers which may not be reproduced in this
electronic version. If you require a printed version of this document
contact the United States Securities and Exchange Commission, Office of
Inspector General, Mail Stop 11-7, 450 Fifth Street N.W., Washington, D.C.
20549 or call (202) 942-4460.
EDGAR UTILITY TO COMMISSION STAFF
Audit No. 351
January 15, 2003
The Electronic Data Gathering, Analysis and Retrieval (EDGAR) system allows Commission staff to electronically search, retrieve, assign and store electronic filings. As an analytical tool, however, EDGAR has limited value to the Commission staff.
We distributed a questionnaire about the utility of EDGAR to a judgment sample of 548 Commission staff; 52 staff responded, for an overall response rate of approximately nine percent. Further, only 44 of the 52 respondents answered our question regarding EDGAR usefulness. While the vast majority of EDGAR users did not respond to the survey, 82% percent of the responses we received regarding our survey question on EDGAR usefulness (36 of 44 total responses to the question) rated the EDGAR system "useful" or" very useful."
We found that the EDGAR system cannot extract financial statement information to automate the selection of filings for review, identify financially troubled companies or analyze financial statement information during reviews. Moreover, it cannot compare original and amended filings to show changes resulting from filing review comments. The Commission requested Commercial Off The Shelf (COTS) software, as part of the EDGAR modernization, to address these limitations. These solutions, however, did not fully meet the needs of the staff because the Commission did not consistently follow IT capital investment acquisition best practices.
We also found data inconsistencies between the Entity Filings and Fee System (EFFS) and EDGAR.
We recommend that EDGAR projects meeting the appropriate criteria should go through the Information Technology Capital Planning Committee's (ITCPC) funding approval process. Also, the ITCPC should ensure that these projects include business process analyses. The Divisions of Corporation Finance (CF) and Investment Management (IM) should request enhanced financial analysis capability. In addition, the Office of Information Technology (OIT) should use performance-based contracting techniques in obtaining EDGAR automation.
OIT should initiate a project, in coordination with CF and IM, to analyze the data in EFFS and EDGAR to identify and correct errors to the extent practicable. Also, OIT and the Office of General Counsel (OGC), in coordination with OFIS, CF and IM, should study the issues related to correcting erroneous filing header information after acceptance and dissemination by the EDGAR system.
During the audit, significant progress has been made in addressing these issues. EDGAR project requests have been submitted to the ITCPC for approval. CF received first-year funding for a financial data system and, in coordination with OIT, substantially completed the EFFS/EDGAR data cleanup effort in October 2002. Also, IM identified a separate financial data system and received first-year funding for it.
SCOPE AND OBJECTIVES
The objective of the audit was to evaluate the utility of the current EDGAR system to Commission staff. We did not evaluate the utility of EDGAR to users external to the Commission (e.g., investors, financial analysts). We surveyed and interviewed staff in the Divisions of Corporation Finance (CF), Investment Management (IM), and Market Regulation (MR), as well as the Offices of Filings and Information Services (OFIS), Information Technology (OIT), and selected field offices. Also, we reviewed and analyzed relevant documents and data related to the EDGAR modernization project and the role of the EFFS.
We conducted the fieldwork between January and May 2002, in accordance with generally accepted government auditing standards.
The Commission's Electronic Data Gathering, Analysis and Retrieval (EDGAR) system receives, stores and distributes electronic filings submitted to the Commission in accordance with securities laws and rules. Because the EDGAR system is the Commission's primary source of corporate filings, it is a critical part of the Full Disclosure program. i
The Office of Filings and Information Services (OFIS), Division of Corporation Finance (CF) and the Division of Investment Management (IM) are the primary EDGAR users. Filer support and fee staffs in OFIS assist electronic filers with their submissions and ensure that the system appropriately calculates and accounts for filing fees. Accountants and attorneys in CF and IM review these filings for compliance with accounting and securities law requirements and upload their review results and related correspondence to the database.
Other Commission staff, however, also use EDGAR. Commission enforcement, inspections and evaluations, and investor education and assistance staffs use the system to retrieve and research historical filings and correspondence.
The EDGAR system includes the EDGAR database and three basic components, or subsystems: receipt and acceptance, analysis and review and dissemination. subsystems. The EDGAR database contains the electronic submissions (filings, correspondence, screening sheets, etc.). The receipt and acceptance subsystem: receives live and test electronic submissions, determines whether the submissions are valid, accepts or rejects (suspends) them, and transmits accepted submissions to the database. The analysis and review subsystem allows managers to assign filings and the staff to query, search, retrieve and upload documents to and from the database. The dissemination sub-system transmits accepted submissions to the contractor who then transmits them to its direct-feed subscribers.ii
The EDGAR system does not send test submissions through its analysis and review or dissemination subsystems. This allows EDGAR filers and the OFIS staff to submit test filings.
The Commission developed EDGAR in three phases: pilot, operational, and modernization. iii The 3-year modernization, initiated in July 1998, introduced several fundamental changes to the system.iv OIT cited work related to Year 2000 system remediation and contractor slippages as the primary reasons for the delay in completing the modernization (it was completed in January 2002). The Commission's estimated cost of the modernization was $22.4 million.
The Office of Information Technology (OIT) recently issued a policy regarding a Commission-wide Enterprise Architecture (EA). vAn EA would describe the relationships between the Commission's primary business activities, data and technologies. This analysis would allow the Commission to evaluate a proposed EDGAR enhancement (or any other information technology proposal) in terms of the data needed, why it was needed, who needed it, where it would be needed, how to process it and when.
The Commission staff that we contacted generally believed that the EDGAR system was useful for routing, researching and assigning filings. Moreover, EDGAR significantly enhanced the flow of most corporate information to the Commission as well as to investors, the public, the information dissemination industry and the securities markets.
The EDGAR system, however, was not designed to extract and analyze financial statement information. Also, while it allows marking of text, it cannot automatically compare original and amended filings to show changes resulting from filing review comments. Data inconsistencies between the Entity Filings and Fee System (EFFS) and EDGAR were also found.
ROUTING, RESEARCHING, AND ASSIGNING FILINGS
We distributed a questionnaire regarding EDGAR utility to a judgment sample of 548 Commission staff. Of the 548 requests, 52 staff responded, an overall response rate of approximately nine percent. vi Of the 52 responses, 44 responded to our question on EDGAR usefulness. While the vast majority of the EDGAR users in our sample did not respond to the survey, 82% percent of the responses we received on our survey question regarding EDGAR usefulness (36 of 44 total responses to the question) rated the EDGAR system "useful" or" very useful."
We were informed that EDGAR provided access to corporate information within minutes, if not seconds, after it accepts the electronic submission. OFIS staff members assign and route new companies to an office or division by Standard Industrial Classification Codes (SIC Code). Subsequent filings from that company are sent automatically to the assigned organization. To assign filings to specific staff to review, managers transmit filings from their electronic inbox to that of the individual reviewer. In the past, reviewers waited hours, or even days, to receive paper filings.vii
During filing reviews, EDGAR facilitates research of historical filings, allowing staff to perform queries based on single or multiple search criteria. Upon completing their reviews, staff may upload their review reports and related correspondence to the database for future reference. In addition, EDGAR allows managers to track the status of filings assigned to staff.
ANALYSIS OF FINANCIAL DATA AND FILING DRAFTS
As noted above, EDGAR is generally considered useful for gathering and retrieving filings. The Commission had limited success, however, in obtaining financial analysis and document comparison capabilities in connection with the EDGAR system. The primary cause of these unsuccessful attempts was that the Commission did not always follow information technology (IT) capital investment acquisition "best practices.viii
Commission staff cannot use EDGAR to identify financially troubled companies, automatically select filings for review or provide ad hoc or pre-programmed analyses to meet staff needs. While most of the respondents to our survey found EDGAR useful, they nevertheless used other sources, such as Compustatix for financial analysis.
During the modernization phase, the contractor proposed a "Financial Data Subsystem" that would include software on staff workstations to access, extract and analyze data from an external financial databasex After some analysis of the proposal, Commission management concluded that its requirements were met by the proposal and funded the project. Later, however, users identified important limitations to the proposal and the project was put on hold.xi To obtain financial analysis capability for EDGAR, the Commission must incur additional costs.
CF and IM identified alternative financial analysis solutions to meet their needs. Both CF and IM received first-year funding (FY 2003) for their financial analysis solutions.
Analysis of Changes to Filings
EDGAR allows filers to mark changes in their filings in response to Commission filing comments (redliningxii. Commission filing review staff expressed dissatisfaction with redlining because of its limitations (in ASCII documents, filers must redline entire paragraphs instead of specific information). As a result, reviewers often obtained marked-up paper copies of filings from issuers. Because reviewers must still manually find the marked-up data within the filing and compare it with their comments and the initial filings, using redlined or marked-up paper filings increased the time needed for filing review and comment follow up.
The Commission expected the modernization project to include the ability to compare the original filing and subsequent amendments, and adequate printing capabilities. Again the Commission was disappointedxiii
Causes of EDGAR's Functional Limitations
Capital investment "best practices," if followed consistently, would have generated better information to use in meeting the needs of Commission staff. These best practices included:
- Business Process Analyses
- User Input
- Performance based contract provisions
--- IT Acquisition Legal Requirements
Several laws and regulations describe IT acquisition best practices.xiv Federal Acquisition Regulation Subpart 37.6 and the Office of Federal Procurement Policy provide guidance on the use of performance-based contract provisions.
--- Business Process Analysis
The Commission was required to analyze and, based on the results of the analyses, revise its processes before making significant investments in technology.xv Also, the federal government's policy on IT investment encourages business process analyses.xvi In addition, the General Accounting Office (GAO) has pointed out the usefulness of business process analyses.xvii The EDGAR contractor proposed an "enterprise and technology analysis" to identify the processes impacted by EDGAR, clarify user requirements and ensure that the system met user needs.
The Commission decided to not perform a business process reengineering, primarily because: the requirements for the previous EDGAR system already existed; the Commission was coming up on the end of the current EDGAR contract and extensions; the equipment was failing and the analysis would have imposed an unreasonable delay. Also, CF pointed out that many existing EDGAR and EFFS functions already met users' needs, and wanted the modernized system to replicate existing functionality.
Because business processes were not analyzed and revised, the requirements developed for the modernization were based on existing processes. The Commission did not take advantage of the opportunity to develop information on its current processes or analyze and revise them before developing its requirements and implementing the modernization. Consequently, to implement process improvements, the staff must submit Program Change Requests (PCRs) or Technical Instructions (TI's) to obtain the appropriate EDGAR modifications.xviii
OIT recently issued an Enterprise Architecture (EA) policy. The EA, when developed, would provide a framework within which the divisions could better link their technology requests to their data and business processes.
--- User Input
EDGAR program officials indicated that there was substantial user input for the EDGAR modernization. Top managers in CF, IM, the Office of General Counsel (OGC), the Office of the Executive Director's (OED) and OFIS were involved in the modernization from the beginning. Also, users from these organizations were involved in writing the request for proposals, evaluating the contractor's proposal, and participating in user groups to develop requirements and evaluate prototypes. In addition, EDGAR Requirements Subcommittee (ERS) meetings provided users with updates on EDGAR issues and a forum to discuss and prioritize changes to the system.
Until recently, however, the ERS only included representatives from OIT, OFIS, CF and IM. Other offices, such as the Divisions of Market Regulation, Enforcement, the Offices of Investor Education and Assistance, Economic Analysis or the field offices, were not represented on the ERS. xix
As a result, OIT and the contractor did not always receive sufficient information on staff needs from these offices. For example, during the modernization, the contractor requested guidance from OIT on use of CUSIP numbers in EDGAR. xx The Office of Economic Analysis often uses CUSIP numbers in analyzing securities industry data, but it appears that it was not contacted. In fact, the OEA staff we spoke with indicated that the lack of usable CUSIP data was one of the primary reasons for their limited use of the EDGAR system. Also, the Division of Market Regulation expressed concerns with the modernized EDGAR's use of broker-dealer data from the Central Registration Depository (CRD) system maintained by the National Association of Securities Dealers (NASD). EDGAR program officials indicated that they were not initially notified of MR's concerns, and would have addressed them had they been notified.
--- Performance Based Contract Provisions
Performance-based service contracting (PBSC) techniques allow contractors freedom to innovate to determine how to meet agencies' requirements, while ensuring that agencies pay only for services that meet their requirements. In order to facilitate objective assessments of contractor performance, performance-based contracts focus on defining the desired results instead of the process the contractor should follow. Federal acquisition requirements identified performance-based contracting as the preferred method of acquiring services.xxi
The Commission did not use performance-based contracting techniques in developing the EDGAR modernization requirements. In performance based contracting, the Commission and the contractor would have to clearly define and agree upon the performance that would result in payment or withholding of financial incentives. A performance-based contract might not have addressed the issues related to the acceptance of the financial analysis system or the word-processing solution for document comparison. A performance-based contract, however, would have allowed the Commission to possibly award or withhold incentive payments to the contractor to the extent that it did not meet established performance goals.
IT Project Approval Processes
In order to improve its acquisition of IT capital investments, the Commission established an IT Capital Planning Committee (ITCPC), composed of senior Commission managers. It subsequently launched the Information Officers Council (IOC), consisting of a senior program manager from each division and office, to support the ITCPC.
The ITCPC issued its first call for projects in FY 1999. At this point, the EDGAR modernization had already begun. At the time, EDGAR projects were requested through the EDGAR Requirements Subcommittee and approved for funding through the EDGAR Steering Committee.
EDGAR projects would benefit from the IT Capital Planning Committee funding approval process. This process would subject EDGAR projects to a formalized process including clear requirements definition, business process analysis and adequate user participation. Moreover, using a single process for all IT projects minimizes duplicative Commission IT acquisition and project management processes.xxii
CF and IM have submitted a number of EDGAR related projects to the ITCPC for consideration. Also, CF and IM selected COTS financial data analysis packages and both received funding for the first year for their financial analysis systems.
EDGAR included inaccurate data resulting from importing incomplete EFFS records from EDGAR and rejecting EFFS records failing its validity tests.xxiii Also, EDGAR contains uncorrected filer header information errors. Left uncorrected, these errors cause EDGAR to distort the affected companies' filing and review information, send filings to the wrong Commission offices and divisions and disseminate inaccurate information.
The contractor moved incomplete records to EDGAR from EFFS because CF did not initially want to migrate filing review and disposition data from EFFS. When OIT executed a routine procedure on these incomplete records in EDGAR, at CF and IM's request, EDGAR inserted automatic entries into the blank data fields.
Also, EDGAR rejected EFFS records that failed EDGAR's data validity tests. For example, EDGAR will accept only one spelling of a company's name as valid, and will reject records with different spellings of that name. Further, between July 2001 and January 2002, CF staff continued to update electronic filing information in the EFFS system, instead of in EDGAR, after the data feed from EFFS to EDGAR was discontinued. As a result, the EDGAR records were not updated.
In addition, despite its filing receipt and acceptance testing, EDGAR sometimes accepted filings submitted with erroneous header information (the filing itself was not affected by these errors).xxiv These errors were identified by the company or by the Commission's staff.
Commission staff corrected certain kinds of data errors (e.g., filing dates, receipt dates). For more complex corrections, Commission staff submitted Program Change Requests (PCRs) to the contractor through OIT.
When the header data was corrected in this manner (post-acceptance corrections, or PACs), however, EDGAR did not re- route or re-disseminate the filing. Staffs in OIT and the Office of General Counsel (OGC) who are involved with EDGAR indicated that the matter of post-acceptance corrections involved a number of technical, legal and procedural issues.xxv
The Division of Corporation Finance, in coordination with the Office of Information Technology, should analyze the data moved from EFFS to EDGAR during the modernization to ensure that the data in both are complete and accurate.
The Division of Investment Management, in coordination with the Office of Information Technology, should analyze the data moved from EFFS to EDGAR during the modernization to ensure that the data in both are complete and accurate.
The Office of Filings and Information Services, in coordination with the Office of Information Technology, should analyze the data moved from EFFS to EDGAR during the modernization to ensure that the data in both are complete and accurate.
OIT, and OGC, in coordination with OFIS, CF and IM, should study the issues involved in EDGAR post-acceptance corrections and report on how best to maintain the integrity and consistency of the database while minimizing filer burdens.
CF and IM indicated that they researched and resolved most of the data errors between EFFS and EDGAR. Also, as noted above, CF, IM and OFIS submitted PCRs to request correction of data errors. Progress continues to be made on the backlog of these PCRs.
i The mission of the Full Disclosure program is to ensure that investors are provided with material information and to prevent fraud and misrepresentation in the public offering, trading, voting, and tendering of securities (U.S. Securities and Exchange Commission Budget Estimate for Fiscal Year 2002, dated April 2001). In FY 2001, the EDGAR system became the Commission's official record for filings.
ii The contractor, through regulated fees paid by the contractor's direct feed subscribers, funds the operation and maintenance of the dissemination system.
iii During the pilot, initiated in 1984, the Commission tested the EDGAR concept with voluntary electronic filers and developed a separate network for EDGAR. The Commission's non-EDGAR network was called the SEC Office Automation System (SECOA). EDGAR users often had two workstations in their offices, one for EDGAR, the other for SECOA. During the operational phase, initiated about 1993, the Commission phased-in mandated electronic filing by companies and eliminated the dual networks. In 1998, the Commission mandated electronic filing for all corporations, absent a hardship exemption.
iv The 3-year EDGAR modernization contract was effective July 1, 1998. The modernization cost was estimated at $22.4 million. In connection with the modernization, the Commission replaced much of EDGAR's previously custom-coded functions with Commercial Off-the Shelf (COTS) software. Also, the Commission introduced two new document formats for electronic filings, HyperText Markup Language (HTML) and Portable Document Format (PDF). HTML, an Internet document format, allows users to link documents and insert applications into documents. PDF (Portable Document Format) provides documents in their original formats, including pictures. Also, the modernized EDGAR allowed companies to transmit their electronic filings to the Commission over the Internet, and implemented browser technology in the internal EDGAR system. The EDGAR modernization was performed by BDM International, Inc., which was subsequently purchased by TRW Inc., which was recently purchased by Northrop Grumman Corp. (contractor). OIT manages the EDGAR contract.
v SECR 24-1.6, November 25, 2002.
vi We received 52 responses (9.4%), distributed almost evenly between Headquarters (26 responses) and the field offices (22 responses). Sixty percent of the respondents used EDGAR for three years or more. Half of the respondents used it daily (53%), 80% at least weekly. Slightly fewer than half (45%) spent two or more hours on EDGAR per session and slightly more than half (55%) spent less than one hour per session.
vii Before the Commission mandated electronic filing on EDGAR, filers submitted their filings in paper form. As a result, processing, distributing and maintaining paper filings and correspondence was much more time-consuming and laborious. Processing and distributing filings often took days (and even longer during filing "peak" periods for 10-Ks when the Commission received even more paper filings).
Before EDGAR, filers delivered multiple copies of their paper filings to the Commission's filing desk, where the filings were reviewed for compliance with filing requirements and manually stamped with the SEC filing date upon acceptance. After acceptance, the staff manually sorted filings into those to send to a contractor for microfiche or microfilming, and those to deliver around the Commission.
OFIS staff hand-delivered the filings according to distribution lists. The filings were stacked in the offices' and divisions' areas for assignment to the review staff. Historical filings were retrieved from paper files maintained in file cabinets. Even more time was needed to obtain records stored off-site.
Members of the public had to visit one of the Commission's Public Reference Rooms (located at the Commission's Washington, DC headquarters and the regional offices) to review company filings.
viii "...Best practices" are defined as techniques that agencies may use to help detect and avoid problems in the acquisition, management, and administration of contracts..." (A Guide To Best Practices For
Contract Administration, Office of Federal Procurement Policy (OFPP) October 1994)
ix Standard and Poor's Compustat database includes financial and market information on reporting companies.
x EDGAR maintains financial information in a text format and the financial information is not "tagged" for use by the system. Consequently, EDGAR cannot extract and analyze financial data directly from the filings in its database. To address these limitations, the Commission attempted to obtain a COTS financial analysis capability as part of the modernization.
xi The subcommittee responsible for reviewing the contractor's proposal, however, did not ensure that the financial database included investment company (or public utility holding company) data. Consequently, the system was of little use to the Division of Investment Management and was not implemented.
xii The EDGAR system provides a "redlining" function, where the registrant inserts formatting tags at the beginning and the end of the information it wants to mark. The EDGAR system displays the text within the tags in a color different from the rest of the text. The EDGAR system did not satisfactorily implement this function until recently. OIG report No. 326 addressed the redlining issue.
xiii The Commission asked the contractor for a COTS product that would compare two documents on a server, then transmit the results of that comparison to a workstation. OIT indicated that no COTS product existed that could meet the Commission's need as it was stated. The contractor proposed that the Commission use the document compare function of a word processing application. The subcommittee reviewed the proposal and found that it would not handle large complex HTML filings. Nevertheless, the Commission accepted the proposal. CF and IM reviewers rarely, however, use this word processing solution because it does not work well with large filings.
xiv The Clinger-Cohen Act (also known as the "Information Technology Management Reform Act of 1996") (Pub. L. 104-106, Division E); the Government Performance and Results Act of 1993(GPRA); the Paperwork Reduction Act (PRA) of 1980, as amended by the Paperwork Reduction Act of 1995 (44 U.S.C. Chapter 35); and OMB Circular A-130, ("Management of Federal Information Resources"). Also, OIG Report No. 337, "Information Technology Project Management," provides detailed information on IT acquisition requirements.
xv Clinger-Cohen Act, Sec. 5123(5): "...the head of an executive agency shall...(5) analyze the missions of the ...agency and ...revise ...mission-related processes and administrative processes as appropriate before making significant investments in information technology..."
xvi The Office of Management and Budget, in Memorandum #M-97-02, "Funding Information System Investments," provides that "... investments in major information systems...should...support work processes that have been simplified, or otherwise redesigned to reduce costs, improve effectiveness, and maximum use of commercial, off-the-shelf technology..."
xvii Executive Guide: Improving Mission Performance Through Strategic Information Management and Technology. AIMD-94-115 May 1, 1994
xviii The three primary vehicles for replacing missing EDGAR functionality include the Program Change Request (PCR), the System Problem Report (SPR) and Technical Instructions (TI). PCRs generally address enhancements (as new requirements), filer header and system data errors (value PCRs). SPRs address hardware and system processing problems and software "bugs," and are done within the existing contract (PCRs and SPRs, however, are considered separate functional requirements). A TI, on the other hand, is a separate contract (time and materials). Because PCRs and TI's are performed on a time and materials basis, the contractor has little incentive to control the cost of this work.
xix The ERS has since included representatives from Market Regulation, the Office of Financial Management and the Division of Enforcement
xx The (CUSIP) system is owned by the American Bankers Association and operated by Standard & Poor's. The National Association of Securities dealers (NASD) and other organizations use CUSIP numbers to identify U. S. and Canadian securities.
xxi The Federal Acquisition Regulations provides guidance on implementing performance-based contracts (48 C.F.R. Subpart 37.6). Also, the Office of Federal Procurement Policy issued best-practices guidance in implementing performance-based contracting (Best Practices for Performance-Based Contracting, Final Edition, October 1998).
Recently, Section 821 of the National Defense Authorization Act for fiscal year 2001 (P.L. 106-398) established performance-based contracting as the preferred method for acquiring services. This Act listed the order of preference for acquiring services: (1) Firm fixed-price performance-based contract; (2) A non-firm fixed-price, performance-based contract; and (3) A non-performance-based contract.
xxii The OIG is currently reviewing the IT capital investment process as a follow up to an earlier review.
xxiii An important part of the EDGAR modernization included merging EFFS (Entity Filings and Fee System) data into the EDGAR database. Data in the EFFS included company names, file numbers, form types, various review codes and disposition dates, and fee records. The Commission used this information to track the status of filings received and reviewed, fees assessed and paid, and to provide information to various other Commission systems. The link between the EFFS and EDGAR systems was to be disconnected after the data merge was completed, 24 months after the contract award date.
xxiv For example, EDGAR may accept a filing submitted with a header containing the wrong form type (10-K or 10-Q instead of 10-KSB or -QSB). Other header errors may involve Central Index Key (CIK) and SEC file numbers.
xxv These issues include, but are not limited to, potential modifications to EDGAR's receipt and acceptance and dissemination systems to catch errors before acceptance by EDGAR, potential burdens to filers and external disseminators related to correcting header errors, determining responsibility for correcting post-acceptance errors and filer liability.