The Supreme Court struggled yesterday with the issue of how to treat libel cases by those who claim they were misquoted by reporters.

The court heard oral argument in Masson v. New Yorker, in which psychoanalyst Jeffrey Masson claims that New Yorker writer Janet Malcolm invented or deliberately altered quotations in her December 1983 series about Masson's feud with the psychoanalytic establishment.

Malcolm denies fabricating any quotes. The 9th U.S. Circuit Court of Appeals, in upholding the dismissal of Masson's suit, ruled that even if she had, the statements were similar enough to those Masson conceded he had made to defeat any claim that she had injured his reputation.

Masson's lawyer, Charles O. Morgan, is asking the court to overturn that ruling and allow a jury to decide the case.

But Malcolm's attorney, H. Bartow Farr III, told the court that "libel law is not concerned with misquotation for its own sake or with setting journalistic standards."

Rather, he said, the First Amendment protects a writer unless the person who claims he was misquoted can show that the writer knew or "strongly suspected" that "the gist" of what was quoted, as opposed to the actual words, was untrue.

"The inquiry is into whether the writer is knowingly misrepresenting something that was said," Farr told the court. "Simply showing that there may be knowledge of different words is not all you need to prove malice." The court has said that public figures like Masson must show "actual malice" -- knowing or reckless disregard for the truth -- in libel cases.

Some justices seemed uncomfortable with that assertion. Justice Antonin Scalia questioned Farr's distinction between misrepresenting the "gist" of what a person said and changing his words. "I don't see this distinction," he said. "It is knowingly misrepresenting substance to say I said something which I did not say."

Chief Justice William H. Rehnquist told Farr his analysis did not make enough allowance for the idea that "putting words in someone's mouth," rather than merely paraphrasing the statements, "can hit a lot harder."

Scalia agreed, saying that in the latter case he took the statements with a grain of salt, knowing that the writer could be putting his own spin on them. "When you say, 'He said, quote,' that is a big difference. I no longer make allowances for your erroneous judgment."

Farr argued, however, that the "proper inquiry" is to look at the substance of what the person has said, compare the disputed words to those that are conceded, and ask whether that is evidence the reporter changed the overall substance of the person's statements.

He noted that Masson had dropped claims that Malcolm had fabricated certain of his quotes after discovering that she had them on tape.

Had that not been the case, Farr said, Masson "would be saying right now that he is entitled to go to the jury simply based on the denial of those words."

The justices also quizzed Morgan closely about the implications of his argument.

Scalia asked Morgan what the harm would be if had made an outrageous statement -- for example, that a particular ethnic group was little more intelligent than dogs -- but was misquoted as saying they were subhuman.

"If they had put that in accurately, it would have been just as defamatory," he said.

Morgan answered that such a decision should be up to a jury.

He told Justice John Paul Stevens that a misquoted source could sue even if the actual quote would have made him look worse. "I proved you fabricated a quote," he said. "I proved it was defamatory." The measure of damages, he said, should be for the jury to determine.

"Under your rule, aren't you going to create serious disincentives to use direct quotes or even to have a tape recorder?" asked Justice Anthony M. Kennedy.

But Morgan said the case involves knowing or reckless misquotation, not a reporter who negligently slips and misses a few words. "We're not talking about the poor reporter who has to get a quote in a hurry," he said.

"But you are," interjected Justice Sandra Day O'Connor. "You are arguing that every misquote gives rise to an inference of actual malice." That, she suggested, might put a reporter "at risk" if the person being quoted insists he never made the statement and the reporter's tape of the conversation is gar- bled.

"At risk," but not necessarily liable, Morgan responded.

At another point, however, Morgan seemed to concede that a misquotation would not be a proper basis for a libel suit if it were substantially true, leading Kennedy to warn him, "You're giving away your theory."