A closely divided Supreme Court acted anew yesterday to restrict access to the federal judiciary by persons seeking vindication of their constitutional rights.

The justices ruled, 5 to 4, against redress in the federal courts for parents who had made "diligent efforts" for 42 days to secure a hearing in a Texas court to regain their three battered children from state custody.

The majority based the ruling on the doctrine that unless a plaintiff faces great and immediate irreparable injury, federal courts should abstain when a state proceeding is pending.

The courts laid down the doctrine in 1971, in case call Younger vs. Harris, and applied it initially to criminal proceedings. Four years later, the court extended the doctrine to civil proceedings involving important state interests.

The court also has been closing the doors of federal courthouses with other kinds of holdings, such as those that have narrowed the right to sue, or standing.

As a result, an appeal for legislation to reverse the trend was made to the chairmen of the congressional Judiciary committees in an April letter from 13 civil rights, consumer, environmental and citizen groups.

The Texas case began in 1976 in Houston when the state Department of Human Resources was told that John P. Sims had savagely beaten his son Paul, 8, in a public school, and that the child had been beaten the night before by his mother, Mary. Paul was treated in a hospital for 11 days.

Department investigators found that the couple's daughters, then 7 and 9 years old, also had been battered. Acting under laws said by Texas to be similar to those in all other states, the department the next day sought an emergency order naming it temporary custodian of the children.

Without the parents being present or represented by counsel the order was issued at once, on March 26, 1976, by the Juvenile Court of Harris County (Houston). The family home is in adjoining Montgomery County.

Five days later, the parents, apparently making the children nominal coplaintiffs, appeared in the juvenile court to seek to have their children returned.

Subsequently, on April 5, the judge issued a temporary restraining order to continue the department's custody, directed the department to file a "Suit Affecting the Parent-Child Relationship," and transferred the entire case to the family's home county.

There, Justice William H. Rehnquist wrote for the Supreme Court, the parents made no known effort to expedite a hearing. Neither did they appeal the restraining order. Instead, on April 19, they souhgt relief in U.S. District Court. Two months of complex procedural manuevers - in both the federal and state courts - followed.

Finally a panel of three federal judges concluded that there was no good reason to stay out of the case and issued a ruling that dealtwith the merits of the Sims' complaint and with nearly every aspect of child abuse proceeding in Texas. The judges felt that absention under the 19'1 Younger doctrine was unwarranted because the Sims case was "multifaceted," because it involved the custody of children and because it grew out of procedural confusion in the state courts.

The panel - which had given the parents their first court hearing - found that the procedures used by the state to take custody of the Sims children were "patently and flagrantly violative of basic constitutional guarantees." In addition, the panel said, "there is a fundamental right emanating from the Constitution which protects the integrity of the family unit from unwarranted intrusions by the state."

For yesterday's majority, Rehnquist wrote that "the only pertinent inquiry is whether the state proceedings afforded an adequate opportunity" for the Sims couple to raise their claims of constitutional violations.

"Texas law appears to raise no procedural barriers," Rehnquist said. Moreover, the parents had regained custody of the children, under a federal-court order, on May 5, 1976, 42 days after losing them. To be sure, he said, there was confusion, but it was "the predictable byproduct of a new statutory scheme," not bad faith.

His opinion was signed by Chief Justice Warren E. Burger and Justices Byron R. White, Harry A. Blackmun and Lewis F. Powell Jr.

For the dissenters, Justice John Paul Stevens wrote that the abstention doctrine was "simply inapplicable" in the Sims case because, as the lower-court panel had said, the state courts had given the parents no "opportunity to fairly pursue their constitutional claims . . ."

Despite what the panel called "diligent efforts," the parents failed to win "even an opportunity to be heard in a state court" on the constitutionality of the seizure and 42-day detention of their children, Stevens emphasized.

For the federal court to have abstained would have deprived the parents of "any relief at all," Stevens said. "By ordering abstention nonetheless," he argued, the majority is not only extending Younger "beyond its logical bounds," but is also "implicitly sanctioning a deprivation of parental rights without procedural protections which, as the state itself agrees, are constitutionally required."

Signing Stevens' opinion were Justices Potter Stewart, William J. Brennan Jr. and Thurgood Marshall.

The court took other actions.

OBSCENITY

The court overturned, 9 to 0, a New York tribunal's validation of a raid on an "adult" bookstore in which the search warrant had not specified all of the 790 items that were seized, and in which the town judge accompanied police and "glanced through" the items before they were hauled away.

The Fourth Amendment to the Constitution does not "countenance open-ended warrants, to be completed while a search is being carried out and items seized or after the seizure has been carried out," Chief Justice Burger wrote.

In a related action, the court agreed to review, in the term starting in October, a ruling by a divided 5th U.S. Circuit Court of Appeals that struck down substantial parts of the Texas obscenity laws. In one of 20 consolidated cases, the appeals court overturned the use of a nuisance-abatement law to shut down an adult movie theater in San Angelo.

Judgeship

The court preserved a ruling that former San Antonio, Tex., prosecutor Ted Butler and Police Chief Emil Peters acted in bad faith when they repeatedly arrested - but never tried to indict - the manager of an "adult" theater for possession of a "criminal instrument": the projector used to screen "Deep Throat." Butler recently withdrew his name from White House consideration for a federal judgeship. He had been recommended by Sen. Lloyd Bentsen (D-Tex.).