A Silver Spring-based subsidiary of ITT Corp. that supplies most of the federal government's electronic mail services programmed its computers to overcharge the government $680,000 over a period of several years, according to an internal federal audit and ITT officials.

ITT/Dialcom Inc., a pioneering computer firm whose clients include the White House and the House of Representatives, also overcharged more than 100 of its private customers by $680,000 in the same period, ITT officials said yesterday.

ITT, which acquired the company, formerly known as Dialcom Inc., in December 1982, said yesterday the firm reimbursed all its commercial customers earlier this year. It has also offered to pay the federal government $1.1 million to cover the overcharges, interest and the cost of a General Services Administration inspector general's investigation into the matter.

Federal officials familiar with the matter said yesterday it represented one of the most sophisticated computer cases encountered by federal investigators. The overbillings, which already have been the focus of separate audits by ITT and GSA, are now the subject of a Justice Department criminal investigation, ITT officials said.

"Yes, it was deliberate," said Howard A. White, ITT Communications general counsel and chairman of ITT/Dialcom, citing the results of the ITT audit. "But we found the problem and . . . we reported it to the government. We have been completely cooperative."

Officials who had responsibility for the computer billings resigned, some on request, White and other ITT officials said.

The $680,000 in overcharges, which took place primarily between 1979 and 1983, represented 38 percent of the company's total billings under its electronic mail contract with the federal government.

Robert F. Ryan, the firm's president, was placed on administrative leave with pay late last year, ITT officials said. Barry Levine, a lawyer for Ryan, said yesterday that there have been no allegations of wrongdoing against Ryan and that his departure had nothing to do with the billings problem. "As far as we know, there was no intent by anyone to overcharge," said Levine. "I do know there were some errors in billing . . . . There were undercharges in some instances and there were overcharges in other instances, both of which were unknown to Mr. Ryan."

The dispute over the billings involves computerized time-sharing services that ITT/Dialcom has provided to more than a dozen federal agencies, including the Department of Agriculture, the Food and Drug Administration, the Army Corps of Engineers, the Federal Bureau of Investigation and the Secret Service.

In a typical instance, an officer in an Army Corps of Engineers office in Oregon can sign on to a computer and type in a brief message that is then transmitted into an ITT/Dialcom computerized "mail bin" in Silver Spring. That message can then be called up on terminals in the corps' Washington headquarters. The General Services Administration, which administers the contract, is billed for the minutes of computer time used.

According to ITT officials and an internal ITT audit, one reason for the overcharges was that federal employes were improperly using ITT/Dialcom's network to play computer games, call up airline schedules and for other uses that were not billable under the company's contract with GSA.

To recover those costs, Dialcom adjusted its computer to overbill GSA for authorized uses by government officials, White said. For example, GSA would be automatically billed for a minimum five minutes of computer time per use when the ITT/Dialcom contract only allowed a three minute minimum.

In addition, the company billed GSA for long distance computer uses at substantially higher rates -- between $3.50 to $6 per hour -- than the $1 per hour it was charging its private customers, according to the ITT audit.

"It was fairly sophisticated," said Thomas Sheehan, a former GSA official who supervised the investigation. "When you consider the amount of services they were providing, think about what it would cost to review every single bill."

ITT, which has been negotiating with lawyers at the Justice Department's civil division and with GSA about a civil settlement of the case, has asked for a written guarantee against debarment of ITT/Dialcom. But some GSA officials have questioned whether ITT may have had knowledge about problems in Dialcom's billings months before it first approached the government with information about the overcharges in November 1983.

Brian L. Astle, a former Dialcom employe who unsuccessfully sued ITT for wrongful discharge unrelated to the billings matter, said in an October 1983 court deposition that he first advised an ITT official about apparent overbillings over lunch in February of that year.

The ITT official said yesterday that "at no time did he Astle mention overbillings."

Edwin A. Kilburn, an ITT vice president, said yesterday that Astle's deposition in October was one of the factors that made ITT realize the seriousness of the problem and report it to the government. But, he added, "there was never any question that we were going to report this to GSA.