An obscure provision of a 1981 Medicaid law has become the focus of a debate over the best way to trim the costs of caring for thousands of low-income elderly and mentally retarded people in nursing homes.

The amendment was designed to encourage people to be treated at home, by providing a wide range of home-health and community services that Medicaid had not previously covered.

But critics charge that the regulations to implement the program, which the Health and Human Services Department recently published in final form, will serve only to limit the number of people who can benefit from the program.

Rep. Henry A. Waxman (D-Calif.), chairman of the House Energy and Commerce subcommittee on health and cosponsor of the provision, said: "The regulations are absolutely too restrictive. The Office of Management and Budget has been pressuring HHS, and these rules are subverting the whole intent. The states are unhappy. We're unhappy. They've tightened it in such a way that they make it difficult to serve people at home."

"It appears the OMB had a heavy hand in narrowing the potential scope of the program," said David P. Racine, legislative chief of the American Public Welfare Association (APWA), which represents state welfare and Medicaid administrators.

Joan Kutcher of the Washington office of New York state said her state also considers the rules too restrictive and may ask Congress to take corrective action.

Neither OMB nor HHS officials would comment publicly on the criticism. But privately they expressed fears that even if a new home-health program were designed to help only those who would otherwise have to live in a nursing home, in practice large numbers of people who previously received no benefits would suddenly come "out of the woodwork" and apply for them.

The officials say that providing the new benefits to people who do not need them would simply create a new federal medical expense, rather than replace costly nursing-home care. And they say that the new benefits would just substitute government money for the home care now provided free by relatives and friends.

Even so, there is agreement that if some of these problems can be avoided, the goal of keeping people out of nursing homes is a good one, for the money that might be saved and because most people would rather remain at home than live in an institution.

Currently, nearly half of all Medicaid funds are spent for nursing-home care. In fiscal 1983, the federal and state governments spent $14.1 billion under Medicaid to provide nursing-home care to 1.5 million people.

More than a third of that amount covered 800,000 elderly people in long-term care facilities. The rest was split equally between facilities for the mentally retarded and skilled nursing homes for people requiring more care.

Since the U.S. population is aging, experts predict that there will be a big upsurge over the next half century in the number of elderly people who need long-term nursing care -- unless more people can be cared for at home.

Under the 1981 Medicaid amendment, any person eligible for Medicaid could qualify for a broad array of new home-based services if the tate judges that he would otherwise have to enter a nursing home.

Among the services that could be included are adult day care, transportation, housekeeping and nursing help. But for the first time, the payments also could cover the cost of installing ramps, handrails and wider doorways and making other modifications to a home; training mentally retarded persons to care for themselves at home; providing help to give family members a break from caring for the beneficiary, and providing a nurse or social worker who could supervise the person's care.

Fearing the "woodwork effect," Congress imposed two major restrictions on the program. A state could not provide the benefits unless HHS agreed to waive normal Medicaid rules. The state must also be able to demonstrate that if the waiver is granted, its per capita costs of care would not be any higher than if the waiver were not granted.

Waivers last three years and can be renewed. The waiver can cover a person, a group, part of a state or the whole state. By restricting the program through the waiver requirement, officials said, the government would be able to test the program without giving every state a blank check to provide the extra services.

As of May 6, HHS had received 165 waiver requests from 47 states and, applying interim rules, had granted 97 of them. Many involved substantial numbers of people.

But the states are still concerned that HHS' standards are too tough.

Racine of the APWA said one problem is that the regulations require a state to give documented estimates of how many nursing beds would be added if it did not get its waiver, rather than simply estimating the number of people who need nursing-home care. He said it will be extremely difficult for many states to document nursing-home construction plans far in advance.

More important, he said, the need for nursing-home care in many states may far exceed the number of beds that exist or are planned. But the number of people who could qualify for home care would, in effect, be limited to the total of existing and planned beds.

In addition, Racine said, the regulations impose a ceiling on total spending for home and community care under each waiver. The ceiling is based on the number of expected beneficiaries multiplied by the estimated cost per person.

Once the waiver is approved, he said, the state is locked into that total, no matter whether more people apply or whether individual patient costs are higher. Thus, he said, states might have to reduce the number of people they can serve, even if the costs of serving them would be lower than if they were in a nursing home.

The states, he said, are particularly concerned because many planned to use the new program to make sure that retarded people who have left institutions continue to get adequate services.