Nine years ago Rep. Robert H. Michel (R-Ill.) pushed through a House provision penalizing states for payment errors in their Medicaid and welfare programs. Because of the long-controversial Michel amendment, 49 states face penalties totaling at least $3 billion by 1989.

Yesterday, a National Research Council panel, after a study mandated by Congress in 1986 under pressure from states, recommended overhauling the penalty system.

The panel said the system should not focus solely on saving money by squeezing the states when they make overpayments but should seek to improve the quality and accuracy of all payments by also punishing states for underpayments or for failure to pay eligible people.

The panel, headed by John C. Netter of the University of Georgia's College of Business Administration, said inaccuracy of payments in the Aid to Families With Dependent Children, food stamp and Medicaid programs should be measured by taking a weighted average of four types of errors:

"Payments to ineligible recipients, overpayments to eligible recipients, underpayments to eligible recipients, and payments that should have been made to applicants or recipients who were wrongfully denied or terminated."

Otherwise, the panel indicated, the penalty system would simply be seen as a way of cutting costs, not as making the programs work as intended.

The federal government has a large stake in reducing errors. Where individuals are underpaid, it means that Congress' aim of providing help to specified groups of the needy is not being carried out. Where they are overpaid, it means that the federal goverment wastes billions of dollars, since it reimburses the states for more than half the assistance in the AFDC and Medicaid programs and for all benefits in the food stamp program, which is also under a similar penalty system. Federal outlays for the three programs in 1987 are estimated to have been $47 billion.

In two other major recommnendations, the panel said the penalty system should consider making allowances for states with caseloads that include a high proportion of exceptionally error-prone cases. For example, the panel said where welfare recipients have some earnings or, as common with low- income elderly, have some assets, their cases are more complex and this should be taken into account in grading performance.

The panel also said that a "quality control system that finds almost all of its operating units to be performing poorly" may be one in which the standards "are too stringent."

Therefore it recommended that a future system set thresholds for penalties or good-performance bonuses "so that only states at the extreme of good or bad performance are subject to rewards or sanctions."

Under existing provisions and regulations, a state is penalized if the proportion of dollars overpaid to recipients or paid to ineligible recipients exceeds 3 percent of its total AFDC or Medicaid payments or 5 percent of its food stamp payments. Since 1986, when the study was ordered, penalties in the AFDC program have been suspended pending completion of the study.

The panel said overpayment error rates had declined to 6 percent by 1984 for AFDC, 8.6 percent for food stamps and 2.7 percent for Medicaid, but those are averages and some states were over.

And the panel recommended clearing away the backlog of penalties, perhaps by penalizing only states with error rates well beyond the national average.