In 1977, the Health and Human Services Department heard that a California doctor had overcharged the Medicare program by more than $130,000 for patients who said they never requested--much less received--his services.

The inspector general's office at HHS did little with the case for three years, then finally referred it to the Justice Department for possible prosecution. But Justice officials decided last year that there wasn't enough evidence to bring charges against the doctor.

This sequence of events is far from unusual. In 1980, the HHS inspector general referred 41 cases of suspected fraud involving doctors, nursing homes, laboratories and other medical providers to the Justice Department. But Justice obtained convictions in only five of the cases, and the longest sentence that any defendant received was five months in jail.

Justice officials decided not to proceed with 31 of the 41 cases, saying they were too old, involved too little money, not enough evidence, or simply lacked what they call "jury appeal." Of the remaining cases, three are still pending, one resulted in an acquittal, and the status of one could not be determined.

While some departments have concentrated on recovering federal funds through civil procedures, HHS recovered money in only four of the 31 cases that Justice rejected. At the end of 1980, in fact, HHS had a backlog of unresolved audits involving nearly $70 million, some of which had been outstanding for more than two years. Under the Reagan administration, that figure has grown to $104 million.

President Reagan praised the inspectors general on Monday for pursuing government waste and fraud with the fervor of junkyard dogs, but some observers say their bark may be worse than their bite. While recent figures suggest that the number of cases being sent to Justice is on the rise in the new administration, it is too soon to determine whether this will lead to more indictments and convictions.

Congressional critics say HHS's track record, at least during the Carter years, has given medical providers little concern that they actually will be prosecuted or jailed for Medicare fraud.

Reagan's new inspector general at HHS, Richard P. Kusserow, plans to respond to these criticisms at a Senate hearing today, a department spokesman said. The hearing is being held by the Senate Aging Committee chaired by John Heinz (R-Pa.), and the Senate Finance Committee, headed by Robert J. Dole (R-Kan.). The spokesman said the department would not comment on the investigations before the hearing.

Some of the cases brought by Justice indicate that physicians often escape with relatively minor penalties. An Illinois podiatrist charged the government for $13,000 worth of foot surgery, for example, when he actually was trimming toenails and removing calluses. He pleaded guilty, was placed on probation for three years and had to repay $5,592.

The toughest sentence was given to an Oklahoma nursing home official, who pled guilty to falsifying 39 monthly cost reports to the government. This official was sentenced to five months in jail, fined $25,000 and ordered to repay $161,000.

Among the cases that Justice dropped was one involving a nursing home official in Washington state who was accused of accepting at least $25,000 in kickbacks from a meat supplier. Justice officials said they could not calculate the exact loss to the government.

Timely enforcement also was a problem for the government. In a third of the unsuccessful cases, more than two years elapsed from the time HHS began to investigate them to when Justice dropped the case.

An official with the Senate Aging Committee said that investigative efforts at HHS generally have been fragmented among several divisions, and that the inspector general's office now has fewer field investigators than the state Medicaid fraud unit in New York alone.