A U.S. Court of Appeals panel said yesterday that an Australian scientist who contended he was libeled in a 1980 article in Science magazine may be entitled to have a jury hear his case.

In an opinion written by Judge Robert H. Bork, the three-member panel said U.S. District Judge Barrington D. Parker erred in dismissing before trial a suit brought by William G. McBride, who contended that he was defamed in an article in the magazine about the controversial "morning sickness" drug Bendectin.

The article, titled "How Safe is Bendectin?," said McBride was paid $5,000 a day for his testimony as an expert witness on the drug, while he contends that that sum, in fact, included compensation for more than just that day's work, according to the opinion.

McBride, an expert in teratology--the study of agents that can cause developmental abnormalities in embryos--had testified on behalf of a plaintiff in a suit brought against Merrell Dow and Pharmaceuticals Inc., manufacturer of the drug.

Bork, joined in the opinion by Judges J. Skelly Wright and George E. MacKinnon, wrote that Judge Parker correctly dismissed two other allegations by McBride. But the panel said Parker should reconsider whether a jury should hear the case because "it is not beyond doubt that a reasonable person might read the article as conveying a defamatory falsehood."

The panel, with some reluctance, said Judge Parker also should consider whether McBride is a public figure, which would mean he would have to prove that the magazine acted with malice and reckless disregard for the truth in printing the article.

"We are troubled by litigation such as this, however," Bork wrote, suggesting that Parker "proceed . . . in a manner that will minimize . . . the burden a possibly meritless claim is capable of imposing upon free and vigorous journalism."

An attorney for the American Association for the Advancement of Science, which publishes the magazine, said he did not know whether the decision would be appealed.