The Supreme Court, acting amidst a nationwide wave of book censorship controversies, yesterday allowed a local school board to be put on trial in federal court for banning books from its school libraries.

The 5-to-4 decision implicitly rejected the argument that federal judges should stay out of these controversies, which would give school boards an untrammeled right to control the content of their libraries.

The vote was badly fragmented, however. Four justices said the removal of the books touches on the students' First Amendment rights of free speech. Four said it does not. And one, Justice Byron R. White, said that crucial question should await a District Court trial.

Thus, the court failed to provide any guidelines for judges or school boards or any assurance that it would rule the same way the next time.

The ruling nevertheless provoked strong response. The decision "symbolizes a debilitating encroachment upon the institutions of a free people," said Justice Lewis F. Powell Jr., the former president of the Richmond School Board and one of the dissenters.

"After today's decision," he said, "any junior high school student, by instituting a suit against a school board or teacher, may invite a judge to overrule an educational decision by the official body designated by the people to operate the schools."

The case, Board of Education, Island Trees Union Free School District No. 26 et al v. Pico, et al., stemmed from a suit against a Long Island, N.Y., school district. In 1975, after several school board members had obtained a list of "objectionable" books at a conservative education conference, the board removed nine books from high school and junior high libraries, describing them as "anti-American, anti-Christian, anti-Semitic and just plain filthy."

The books were: "A Hero Ain't Nothin' But a Sandwich," by Alice Childress; "A Reader for Writers, A Critical Anthology of Prose Readings," compiled by Jerome Archer; "The Best Short Stories by Negro Writers," compiled by Langston Hughes; "Down These Mean Streets," by Piri Thomas; "The Fixer," by Bernard Malamud; "Go Ask Alice," author anonymous; "The Naked Ape," by Desmond Morris; "Slaughterhouse Five," by Kurt Vonnegut Jr., and "Soul on Ice," by Eldridge Cleaver.

Yesterday's suit was brought by five students, represented by the American Civil Liberties Union, who sued under federal civil rights law to have the books restored. It was one of several similar suits brought around the country.

A U.S. District Court judge in New York threw out the suit, but the 2nd U.S. Circuit Court of Appeals reversed, saying First Amendment free speech guarantees were implicated by removal of the books. The appellate court ordered a full trial to determine whether those guarantees were violated. The school board appealed.

Yesterday five justices said the trial should go on, but one of them, White, declined to discuss the crucial First Amendment issue. He said the court could do that, if necessary, after a trial determined the full circumstances of the Island Trees book removals. "If, for example, the District Court concludes after a trial that the books were removed for their vulgarity, there may be no appeal," White said.

Justice William J. Brennan Jr., joined by Justices Thurgood Marshall and John Paul Stevens, took the broadest approach, declaring a "right to receive" information for the students that prohibits library censorship of ideas and opinions. That right, he said, "is an inherent corollary of the rights of free speech and press that are explicitly guaranteed by the Constitution."

It is permissible to censor for vulgarity, Brennan said, but "our Constitution does not permit the official suppression of ideas . . . . If the school board members intended by their removal decision" to suppress ideas just because they disagree with them, the banning is impermissible. That intent, he said, is what the trial should consider.

Justice Harry A. Blackmun agreed for different reasons that the students may invoke the First Amendment. He said it is not just suppression of ideas that may be involved but "discrimination between ideas . . . . I find crucial the state's decision to single out an idea for disapproval and then deny access to it," he said.

" . . . . We strike a proper balance here by holding that school officials may not remove books for the purpose of restricting access to the political ideas or social perspectives discussed in them."

Chief Justice Warren E. Burger, in a dissent joined by Justices Powell, William H. Rehnquist and Sandra Day O'Connor, called the Brennan opinion "a lavish expansion going beyond any prior holding under the First Amendment . . . . Were this to become the law, this court would come perilously close to becoming a 'super censor' of school board library decisions."

Rehnquist, joined by Burger and Powell, also wrote separately as did O'Connor.

Book banning disputes have been staples of political controversy for years. But in the past two years, attempts to screen libraries have increased fivefold, to the dismay of the American Civil Liberties Union, the American Library Association and other civil libertarians. Most of the attempts have come from conservative and fundamentalist quarters, but they have not monopolized the field. Recently, a contingent of blacks sought removal of Mark Twain's classic "Huckleberry Finn" from the curriculum of the Mark Twain Intermediate School in Fairfax County, Virginia, saying it contained racist stereotypes.

Yesterday's decision did not deal with curriculum screening, though Brennan said he thought school boards have greater flexibility in that area.

Steven Pico, one of the students who originally challenged the board's ban on the books, called the opinion "a victory, though not an absolute one," saying it gives "a green light to go to court and tells a board it can't ban books if its prime motivation is political."

Judith Krug, head of the American Library Association's intellectual freedom committee in Chicago, said she is pleased because "we preserved what we wanted most, access to the court system when all else fails."

The president of the Island Trees school board, Frank A. Martin, a retired New York City police officer, said the decision made it "a sad day for the parents, the children, and the public schools of America. . . . In effect the decision nullifies local control."

George W. Lipp Jr., the board's attorney, said he was "very disappointed that the court gave us no guidelines with regard to what is permissible and what is not."

Gwen Gregory, deputy legal counsel for the National School Boards Association, which declined to support the Island Trees board in the case, said the decision "is the most confusing I've read."