The Supreme Court yesterday opened newsroom editing processes to inquiry by "public figures" seeking important evidence for libel suits against the press.

In a 6-to-3 ruling, the court held that the First Amendment does not give members of the press an absolute privilege to refuse to answer questions about their "state of mind" when they circulated erroneous information that damaged a person's reputation.

Justice Byron R. White wrote the opinion of the court. Justices William J. Brennan Jr., Potter Stewart and Thurgood Marshall each filed a dissenting opinion.

The ruling came in case in which retired Army officer Anthony Herbert accused the Columbia Broadcasting System, producer Barry Lando and commentator Mike Wallace of having falsely and maliciously defamed him, in a 1973 "60 Minutes" television program, by casting doubt on his charges that superior officers covered up reports war crimes in Vietnam.

He filed a $44.7 million libel action in which he acknowledges that he was a "public figure." As such, he has the burden, under the court's 1964 New York Times vs. Sullivan decision of proving that the defendants had aired a damaging falsehood "with 'actual malice'-that is, with knowledge that it was false or with reckless disregard of whether it was false or not."

In 26 pretrial sessions over the course of a year, Chief Justice Irving R. Kaufman of the Second U.S. Circuit Court of Appeals wrote later, producer Lando answered "innumerable questions" about "what he knews, or had seen; whom he interviewed; intimate details of his discussions with interviewees, and the form and frequency of his communications with sources."

But Lando declined to answer inquiries about his conclusions in the investigatory phase as to persons or leads to pursue; his judgment of the veracity of persons he interviewed and the facts they gave him; his basis for conclusions about the reliability of sources, information, or events; his talks with Mike Wallace about what material to include in or exclude from the broadcast, and his intentions as indicated by his decisions on the material to be used or discarded.

U.S. District Court Judge Charles S. Haight Jr. held that Lando had to answer the inquiries. The producer's state of mind being of "central importance" to the issue of malice, the questions were relevant and "entirely appropriate," he said.

A divided Second Circuit reversed. The answers sought by Herbert "strike to the very heart of the vital human component of the editorial process," Judge Kaufman wrote for the 2-to-1 majority. "Faced with the possibility of such an inquisition, reporters and journalists would be reluctant to express their doubts. Inded, they would be chilled in the very process of thought."

The high court's rejection of this view drew predictable reactions.

Lt. Col. Herbert, reading a statement to reporters, praised the decision, saying, "The issue has been, and is, whether the press has the right to knowingly abuse the privilege it has been granted. . . I believe the Supreme Court has correctly held that I have a right to prove my contentions."

Bill Leonard, president of CBS News, called the ruling "another dangerous invasion of the nation's newsrooms" that goes beyond last year's dental of constitutional protection of journalists' notes and files. It refused "constitutional protection to the journalist's most precious possession-his mind, his thoughts, and his editorial judgment," Leonard said.

For the court, Justice White said that its past rulings, specifically including New York Times, not only didn't suggest any restriction on the sources from which a libel plaintiff can get the evidence he needs to prove the critical elements of his case, but actually made it "essential" that he focus on a defendant's "conduct and state of mind."

In the Herbert case, White said, the court was refusing to modify "firmly established constitutional doctrine" by erecting a new "impenetrable barrier" to direct evidence of a knowing or reckless falsehood. He noted the fear that the ruling, by increasing the costs of litigation for small news enterprises, could lead to "self-censorship," but said that only an-untenable total immunity from liability could cure this problem.

None of the dissenters wanted to affirm the 2nd Circuit in full.

Justice Brennan agreed with the majority that the "editorial privilege" claimed by CBS doesn't shield "factual matters" that may bne sought in pretrial discovery.

He worote, however, thtat the privilege should protect "predecisonal communication among editors" so long as a public-figure plaintiff can't demonstrate that a libel, on its face, "constitutes defamatory falsehood."

Justice Marshall said that the majority was "professing to maintain the accomodation of interests struck in New York Times . . . .'"

But the ruling is "unresponsive to the constitutional consderations underlying that opinion," particularly, he said, its goal of ensuring "uninhibited [and] robust' debate on public issues. . ."

Justice Stewart wrote that as he understands the constitutional rule laid down by New York Times , "inquiry into the broad "editorial process' is simply not relevant in a libel suit brought by a public figure against a publisher. And if such an inquiry is not relevant, it is not permissible."

He emphasized that "malice" as used in the 1964 ruling "simply does not mean malice as that word is commonly understood. In common understanding, malice means ill will or hostility . . ."

But the phrase "actual malice" in the Times case "has nothing to do with hostility or ill will," he said. For that reason, he concluded, the question normally asked o f a person motivated by true malice-"why?"-is "totally irrelevant" to the constitutional standard.

In addition to the dissents, a separate opinion was filed by Justice Lewis F. Powell Jr., a member of the majority, to stress the point that in supervising discovery in a public-figure libel suit, a federal judge "has a duty to consider First Amendment interests as well as the private interests of the plaintiffs." CAPTION: Picture, ANTHONY HERBERT . . . suing CBS for $44.7 million