The Supreme Court agreed yesterday to decide whether a public figure suing for libel can probe the thoughts and opinions of journalists to find out if malice motivated them while they were preparing a story.

The justices granted a petition to review the first holding by a lower tribunal that the Constitution protects editorial deliberations against intrusion by libel plaintiffs.

The petition was filed by Anthony Herbert, a retired Army lieutenant colonel who claimed that the Columbia Broadcasting System defamed him in a 1973 "60 Minutes" program that cast doubt on his charges that Army officers in Vietnam covered up atrocities by American troops.

In another First Amendment case, the court dismissed a petition by a newspaper and a television station in Miami to review a decision upholding a Florida law that prevents reporters from making concealed tape recordings of conversations and phone calls. Three of the nine justices wanted to hear argument in the case.

The court also declined to review a decision in a New York City case prohibiting a high-school newspaper from making and reporting a confidential survey of students' sexual attitudes.

In the background of the Herbert-CBS libel case, which will be argued in the term starting in October, is a 1964 Supreme Court ruling that a plaintiff seeking damages for a defamatory statement must show that the statement was made with "actual malice," that is, "with the knowledge that it was false or with reckless disregard of whether it was false or not . . ."

CBS and its co-defendants, "60 Minutes" producer Barry Lando and commentator Mike Wallace, argue that journalists' state of mind can be inferred from "objective facts."

Herbert contents, "with the knowledge that it was false or with reckless disregard of whether it was false or not . . ."

CBS and its co-defendants, "60 Minutes" producer Barry Lando and commentator Mike Wallace, argue that journalists' state of mind can be interred from "objective facts."

Herbert contends, however, that if the unprecedented holding by a divided 2nd U.S. Circuit Court of Appeals in November is allowed to stand, "practical litigants may well conclude that any remedy for libel against a journalist by a public figure is now illusory."

In the opinion for the 2-to-1 majority, Chief Judge Irving R. Kaufman expressed no view on the merits of the dispute, which is still pending; he dealt entirely with Herbert's claim that he is entitled to inquire "into Lando's thoughts, opinion and conclusions."

"The answers he seeks strike to the heart of the vital human component of the editorial process," Kaufman wrote. "Faced with the possibility of such an inquisition, reporters and journalists would be reluctant to express their doubts. Indeed, they would be chilled in the very process of thoughts."

In a dissent, Judge Thomas J. Meskill scorned the "notion" that a public figure seeking to prove actual malice should be denied the right to inquire into the defendant's mental state.

"The publication of lies should be discouraged," he wrote. "The discovery . . . of an editor's state of mind will not chill First Amendment activity to any greater extent than it already is being chilled" by the 1964 decision.

In reversing a trial judge, Kaufman pointed out that over the course of a year, in 26 sessions producing a 2,900-page transcript, Lando answered "innumerable questions" by Herbert about "what he knew, or had seen; whom he interviewed; intimate details of his discussions with interview; and the form and frequency of his communications with sources."

Praising such "remarkable" cooperation, Kaufman said that his court cannot permit the "invasion of First Amendment rights" represented by Herbert's effort to get answers to "a small number of questions relating to his [Lando's] beliefs, opinions, intent and conclusions in preparing the program."

In the Florida case, the issue was a 1974 law making it a criminal offense for a person to intercept a wire or oral communication except when all of the parties to it have consented.

The Miami Herald and TV station WCKT sued. They protested that the accuracy, candor and corroboration enhanced by concealed recordings makes possible important stories in areas such as corruption, consumer fraud and housing discrimination. The law is an unconstitutional prior restraint of publication, the contended.

A trial court agreed. The Florida Supreme Court dismissed the paper's and the station's appeal "for want of a substantial federal question." The three justices who dissented, William J. Brennan Jr., Byron R. White and Harry A. Blackmun, wanted to hear arguments.