After an initial shrinkage caused by President Reagan's cutbacks, the nation's relief rolls are growing again, confounding administration efforts to reduce the federal welfare bill.

New figures released this week by the Department of Health and Human Services show that 159,000 people were added to the Aid to Families with Dependent Children program last summer. The expansion brought the total number of recipients to 10,346,738 in September, 1982.

While HHS says it has no more recent data, an independent study by a group of state welfare administrators shows that the caseload continued to climb the rest of the year and will rise an additional 4 percent this year.

"Obviously, when unemployment is up and people are out of work, more people are going to be eligible," said Linda S. McMahon, associate commissioner for family assistance at HHS and the chief administration official charged with bringing welfare costs down. "As the economy improves, we expect to see the trend stabilize," she added.

The new figures have revived a long-standing debate over the administration's controversial welfare revisions, designed to cut the caseload, that took effect in October, 1981.

Officials argue that those changes shaved more than $700 million a year from federal welfare spending. But opponents contend that the current upward trend is evidence of a "boomerang effect" resulting from disincentives to work that were built into the president's alterations.

"People went off the rolls and then they gave up whatever jobs they had in order to get back on," says Marjorie Kopp of the Child Welfare League and member of a group called the Ad Hoc AFDC Coalition.

Under the Reagan changes, AFDC eligibility criteria were tightened severely for the so-called "working poor." In computing recipients' benefits, for example, strict new lids were placed on work-related and child care deductions.

The idea was to restrict welfare to the "truly needy," but critics of the changes said the result in some cases was to induce working welfare recipients to quit their jobs rather than have their benefits cut off or reduced. Losing AFDC usually means losing automatic eligibility for Medicaid and food stamps.

In Wisconsin, an estimated 42 percent of recipients who were forced off the rolls because of the working-poor restrictions during the early part of 1982 were back on by last October, says a new Ford Foundation-financed study by the Center for the Study of Social Policy, a non-profit research group.

In Los Angeles County, the AFDC caseload dropped from 181,300 to 173,375 during the first six months of the Reagan program, then shot back up to 184,116 by last November, the study found.

But hard data are hard to come by. McMahon counters that an internal HHS study by a private consultant has found that no more than 10 percent of the caseload affected by the administration's restrictions returned to the rolls. That is "about average for this group in any event....It's not unusual for people to come back on welfare at all," she insisted.

"Those criticisms about the work restrictions have not come true . . . . People are motivated by more than dollars and cents."

The rise in AFDC population has dimmed hopes of reducing federal welfare costs, although administration officials have credited their revisions with slowing the growth of the program. The federal share of AFDC, whose expenses are shared approximately 50-50 with the states, is projected to climb by $200 million this year to $7.7 billion, or about the same as it was when Reagan took office. The Congressional Budget Office estimates that the Washington share will increase to $7.8 billion next year.

The administration's fiscal 1984 budget proposes new cuts, such as mandatory workfare and an end to parents' payments once their children reach age 16. Congressional resistance has been fierce.

Meanwhile, state welfare costs are rising even faster as recipients removed from AFDC rolls transfer to state-financed general assistance programs. In New York, for example, the general assistance caseload rose 25 percent and costs 30 percent, according to the social policy center's study.

"We made a decision at the federal level that the program should be narrowly targeted," said McMahon. "The states have made decisions to support people that we feel should not be supported by the government."