The Reagan administration's controversial "Baby Doe" rule to protect handicapped babies in hospitals went into effect yesterday after a federal judge temporarily rejected pleas from medical groups that it be stopped.

The regulation requires hospitals receiving federal aid to post notices warning that failure to provide food and treatment for newborn infants with serious birth defects is subject to prosecution for violating the infant's civil rights. It also sets up a toll-free hotline to take anonymous complaints from across the country.

U.S. District Court Judge Gerhard A. Gesell declined to issue a temporary restraining order preventing the regulation from going into effect. But he ordered an expedited hearing on the merits of the issue, including whether the Department of Health and Human Services has authority to use laws protecting the handicapped for this purpose.

The American Academy of Pediatrics, the National Association of Children's Hospitals and the Children's Hospital National Medical Center filed suit Friday to block implementation until they had a chance to comment on the rule.

It went into effect 15 days after it was published, without the usual public-comment period of 60 days.

The regulation was prompted by the death of "Baby Doe" in Bloomington, Ind., last spring. The parents, doctor, and state court allowed food and treatment to be withheld from a child born with a condition that causes mental retardation and required surgical intervention.

Health and Human Services Secretary Margaret M. Heckler said yesterday she was "pleased the judge did not see fit to interfere." Stephan Lawton, a Washington attorney for the pediatrics group, said a hearing on the issue had been set for April 8. He added that Gesell indicated "he continues to be concerned about the authority of agencies to establish medical standards."