Some FBI officials began raising doubts about the bureau's attempts to create a computerized case management system as early as 2003, two years before the $170 million project was abandoned altogether, according to a confidential report to the House Appropriations Committee.
By 2004, the report found, the FBI had identified 400 problems with early versions of the troubled software -- but never told the contractor. The bureau also went ahead with a $17 million testing program last December, even though it was clear by then that the software would have to be scrapped, according to the review.
The 32-page report -- prepared by the House committee's Surveys and Investigations staff and obtained by The Washington Post -- indicates that the FBI passed up numerous chances to cut its losses with the doomed Virtual Case File (VCF), instead forging ahead with a system that ultimately cost taxpayers more than $100 million in wasted expenditures.
The report chronicles a list of errors and misjudgments that were made during the software project's troubled history, from assigning underqualified personnel to poor oversight and inadequate planning. And while the bureau has enacted important reforms in its information technology efforts in recent months, one FBI official warned that the bureau remains far behind where it should be.
"We are still crawling," the unidentified official told investigators.
FBI officials said last week that the bureau has undertaken a broad and rapid restructuring that will solve many of the problems outlined in the report. They also said that a new program -- code-named Sentinel -- will rely on off-the-shelf software rather than the kind of custom approach that contributed to many of the problems with VCF.
"To say we're crawling is inaccurate," said one official, who spoke on the condition of anonymity. "We are building a system from the ground up and have made real progress."
The House committee's report is the latest in a series of damning evaluations of the collapse of VCF, which was formally dumped in March after FBI Director Robert S. Mueller III concluded that it was already outdated and did not perform as expected.
The system was part of Trilogy, a $581 million FBI program that includes a new computer network and thousands of new high-speed personal computers for agents and analysts. VCF would have been the final major step in the upgrade, providing a modern database for storing case information and allowing agents to share and search files electronically.
Numerous outside experts and panels have criticized the FBI's paper-based records system as outmoded and inefficient, and the commission that investigated the Sept. 11, 2001, attacks concluded that the shortcomings may have contributed to the failure to detect the al Qaeda plot. The Justice Department's inspector general warned in February that the FBI's continuing technology problems had "national security implications" and that agents were "significantly hampered" in their efforts to prevent terrorism and combat other serious crimes.
The new report, which is not scheduled for public release, reveals that "some officials involved in VCF's development began to see problems" in early 2003, about a year after the FBI and its contractor, Science Applications International Corp., began focusing on creating the case management software.
The problems involved "the progress of the software development efforts" and were "attributed to contracting and program management oversight," according to the report.
More problems -- this time "technical and functional deficiencies" -- became apparent in December 2003, when SAIC delivered its first batch of software to the FBI. By March 2004, the FBI had "identified 400 problems but did not share them with SAIC because it did not want the contractor to think these were the only issues remaining," the report says. About the same time, an official in the bureau's Cyber Division "voiced serious concerns" about the status of the project and recommended an independent review.
These internal debates and concerns stand in contrast to public assurances from FBI officials during the same period. For example, Zalmai Azmi -- the FBI's new information technology chief and the fourth person to occupy the position during the VCF project -- predicted in April 2004 that the first version of VCF would be delivered by the end of the year.
But according to the congressional report, problems continued to mount as the FBI and SAIC feuded over system requirements, changes and other issues. In June, work stopped on the broader VCF software program to focus on a pilot project. By December, when an outside consultant had recommended scrapping the software in favor of commercial products, the FBI nonetheless moved ahead with the pilot program run out of its New Orleans field office.
Several officials interviewed by the House staff questioned the decision to proceed with the expensive pilot test of a product that was headed for the trash. "On no planet I know does it make sense" to spend $17 million on such an effort, one official said. Others contended that "it was done for political reasons because the FBI believed it had to deliver something," according to the report.
FBI officials dispute criticism of the pilot program, which continued through March 2005 and resulted in mostly negative reviews of the software. Azmi said in a statement posted on the FBI's Web site that the exercise provided "critical information . . . that we could use for the next generation of case management."
Agents in the field "told us what they liked, what they didn't like and how to improve," Azmi said. " . . . We got our money's worth out of the evaluation. Now we're charting a new course and look forward to publicizing it when it's ready."
Although the report says the FBI finally decided to abandon the project partly because of "deficiencies in the VCF product delivered," the committee's inquiry focused on FBI shortcomings. SAIC officials have strongly defended their role in developing the software and have complained about frequent FBI management turnover and design changes -- including 36 to the contract alone.
Those complaints are largely confirmed by the House investigation and the earlier inspector general report, which concluded that "the main responsibility for the problems with Trilogy rests with the FBI."
An SAIC spokeswoman declined to comment on the new report because the firm has not seen a copy.
Mueller has said in congressional testimony that the actual loss from VCF will total about $104 million. The rest of the money was either unspent or spent on items that can be used in future applications, he said.
Rep. Frank R. Wolf (R-Va.), who requested the report from the investigative staff, said in an interview that he is hopeful the FBI "has learned a lesson" from the problems with the earlier software system. Wolf, who chairs the appropriations subcommittee that oversees the FBI, also said he has confidence in Mueller and his ability to solve the technology problems.
"I think it validated what a lot of people already suspected, which is that they had some real personnel problems," Wolf said. "They didn't have people who were up to being able to do something this complicated. . . . They'll have to be doubly sure it doesn't take place again."