A Reagan administration task force struggling to redraft a controversial rule on hospital treatment of handicapped newborns has become caught in an unusual political and legal minefield.

Bound by President Reagan's commitment to a strong "Baby Doe" rule, officials from the Justice and Health and Human Services departments are debating whether the rule could broaden civil rights protections in ways the administration would like to avoid.

The officials also are trying to be conciliatory toward doctors' and hospital groups that mounted a successful legal challenge to the administration's first rule on handicapped newborns, while assuring "right-to-life" advocates that the new regulation will be just as tough as the original.

A new version of the rule is expected soon, but final decisions have not been made, government sources say. If the administration appeals the lower court decision as expected, it must submit its brief this week.

At the same time, Congress is moving to revise the Child Abuse Prevention and Treatment Act, seeking to add protections to handicapped newborns in hospitals.

The administration's original regulation required hospitals to post signs warning that failure to feed and treat handicapped newborns is against the law. It was published in March without the customary 60-day public comment period. At the same time, HHS also set up a hotline to take anonymous complaints of violations.

"We're not going to give up," HHS spokesman Claire del Real said. She said the court's procedural objections would be overcome by publishing a new notice of proposed rulemaking and allowing public comment. But there were differing accounts as to whether the substance of the new rule would be different.

U.S. Surgeon General C. Everett Koop, a member of the administration task force, said he thought there would be "a preamble that is more explicit about the intent of the rule" and how it might be applied.

Evan Kemp of the Disability Rights Center said last week that he had seen a draft rule containing an explanatory section that gave examples of the standards the government would want to see applied in cases of Down's Syndrome, a genetic condition causing mental retardation; spina bifida, a spinal defect which can cause retardation or paralysis, and anencephaly, a deadly condition in which crucial parts of the brain are missing or malformed.

Dr. Koop said the "greatest concern" was that Down's Syndrome babies with correctable defects not be "starved to death." In a "Baby Doe" case in Bloomington, Ind., last year, an infant died after its parents and a doctor, with the backing of the state's highest court, asked that food and treatment be withheld.

But there are cases, Koop said, such as anencephalic babies, "where the child should be kept comfortable and treated humanely and let nature take its course."

Medical groups have complained that the government did not specify how the rule would be applied and expressed concern about federal intervention in medical decisions.

Earlier this month, government officials met with representatives of the American Academy of Pediatrics, the American Hospital Association, the American College of Obstetricians and Gynecologists and Children's Hospital here. Participants described the session as "sincere" and "productive."

One person at the meeting reported that HHS Undersecretary John A. Svahn, the chairman of the task force, said the new regulations would be designed to meet both the "concerns of the court" and the "concerns of hospitals and the medical profession." The groups argued that the best approach was to set up individual hospital ethics committees to advise parents and doctors when difficult situations arise.

Task force members also have met with "right-to-life" advocates, who say that nothing short of the "hotline" procedures will suffice. They also feel that, to be effective, the new regulation must reiterate HHS's policy that the receipt of Medicare and Medicaid funds automatically subjects hospitals to such civil rights laws as Section 504 of the Rehabilitation Act of 1973. On this point, the "right-to-life" groups have backing from an unexpected quarter: some liberal civil rights groups such as the Center for Law and Social Policy.

Barbara Milstein, a lawyer with the center, said last week that while she opposes the handicapped baby rule, she thinks Medicare and Medicaid funding should oblige hospitals to abide by civil rights statutes.

While getting support on this narrow issue from their usual opponents, "right-to-life" advocates say they are being opposed by one of the administration's most visible conservatives: William Bradford Reynolds, the assistant attorney general for civil rights, who is also a member of the task force.

"We're very concerned about the historical opposition of the Justice Department and we are concerned that opposition from that quarter is continuing," said Gary Curran of the American Life Lobby.

Reynolds' concern, according to government sources, is that defining Medicare and Medicaid payments as "federal financial assistance" under a civil rights law could open a Pandora's box, forcing the administration to investigate alleged civil rights violations in grocery stores that accept food stamps and colleges whose students get work-study grants.

Reynolds made this argument unsuccessfully at a White House meeting last month, according to one source. Since then, however, he has suggested that the rule ought to have an additional legal justification.

Because states receive child welfare grants from the federal government, Reynolds has told associates, HHS should require the states to promulgate and enforce their own "Baby Doe" rules. If they fail to do so, the state grants should be withheld because the states are not enforcing civil rights protections for the handicapped, under this theory.