In recent years, Supreme Court Justice William J. Brennan Jr. has dissented strongly to decisions that he saw "as hampering, if not shackling the press' performance of its crucial role in helping to maintain our open society."

But in a speech yesterday, he accused some newspapers, broadcasters and press organizations of criticizing those same decisions with unwarranted bitterness, "excessec," "vehemence . . . out of all proportion," and in one case, "unjustified violence."

And he had a warning, delivered in what he called "hard words": either the press, like other insititutions, will recognize that it "must accommodate a variety of important social interests," which "the sad complexity of our society makes . . . inevitable," or it will face "a shrill and impotent isolation."

Much is at satke, including "the ability of the press to resume its sure voice as a reliable conscience of this nation," and its credibility, Bernnan said.

His unusually forceful speech, which named many of his targets, was given at the dedication of the Samuel I. Newhouse Law Center of Rutgers University in Newark, N.J.

Brennan said that the source of press overreactions "stems from the confusion of two distinct models of the role of the press . . . that claim the protection of the First Amendment."

One is the "speech" model. It sees the amendment's primary purpose as being 'more or less absolutely to prohibit any interference with freedom of expression." In other words, the press is free to print information it gets. This is relatively noncontroversial. In the past two years, Brennan noted, the court unanimously struck down state laws barring the press from speaking out on certain subjects. "The 'speech' model thus readily lends itself to the heady rhetoric of absolutism," he said.

But, he said, such rhetoric is "only obfuscatory" when applied to the "structural" model. Under it, he said, the First Amendment "protects the structure of communications necessary for the existence of our democracy." In everyday terms, the press is shielded not only when it publishes the news it gets, but also when it "performs all the myriad tasks necessary for it to gather and disseminate the news."

The problem perceived by Brennan is that "the stretch of this protection is theoretically endless." Consequently, any limits put on the press will affect its ability to perform constitutionally protected functions. The upshot, he said, is that courts must weigh the effects of an inhibition imposed on the press "against the social interests served by the imposition."

Such a balancing occurred in 1972, when the Supreme Court held that First Amendment interests involved in compelling a reporter to reveal confidential notes or sources to a grand jury were outweighed by society's need to enforce criminal laws.

But Brennan's chief case in point was last April's ruling, in Herbert vs. Lando, that a public figure suing for libel could ask a defendant journalist about his state of mind -- his intent -- when he published an alleged defamatory falsehood. The decision produced criticism of "unjustified violence," a "firestorm of acrimonious criticism," he said.

Yet, said Brennan, inquiry into state of mind "is one of the most common procedures in the law . . . And, in the area of libel, it would be scarely fair to say that a plaintiff can only recover if he establishes intentional falsehood and at the same time to say that he cannot inquire into a defendant's intentions."

In dissenting, Brennan urged a "qualified privilege" for the communications that precede final news jedgments. In the speech, he said the "the cedibility of the press was impaired by the excesses of its reactions to Lando." Thus, he said, the press undercut its "quite correct" opposition to the later ruling, in Gannett vs. DePasquale, that the Sixth Amendment gives the public no right to attend pretrial hearings in a criminal case.