In a case that could break new ground, an American newspaper is engaged in a Canadian court battle over what it may publish about news in Canada.

The case involves the conviction in Edmundston, New Brunswick, last month of an American reporter for The Bangor (Maine) Daily News on charges of violating a Canadian judge's gag order.

Although the case directly involves only the northern Maine newspaper, the issue also affects newspapers in such other border cities as Buffalo, Detroit and Seattle. Besides obvious questions of freedom of the press, the case involves how much control the Canadian judicial system can exert over a U.S. corporation.

At the heart of the dispute is a Canadian law that, in a preliminary legal hearing, the defense can ask the judge to ban dissemination of news even though the hearing is open. The law is intended to keep pretrial publicity from prejudicing potential jurors.

"It's a question of whether a Canadian court can reach into the United States to stop an American newspaper from writing a story for American citizens about a Canadian problem," said Jack Landau, director of the Reporters Committee for Freedom of the Press.

Landau also complained that Canada's prosecution of Beurmond Banville, the Bangor reporter, was unfair because Banville neither published nor distributed the newspaper inside Canada.

Only 56 issues of the offending newspaper were sold in all of Canada. Edmundston, across the St. John River from northern Maine, is 180 miles from Bangor.

The Bangor Daily News' problem began in February. Banville appeared at the Edmundston courthouse to cover a preliminary hearing for Samuel Gagnon, 21, since convicted of the rape, beating and murder last New Year's Eve of an ill, disabled 80-year-old widow.

The crime was widely publicized and received extensive attention from the area's Canadian and U.S. newspapers. While Banville was in the courtroom, the judge banned press coverage of the case.

Of The Bangor Daily News' daily circulation of about 83,000, only 56 copies are sold in Canada. Banville is chief of the paper's bureau in Madawaska, Maine, just across the St. John River from Edmundston.

Managing editor Paul Reynolds said that Banville returned to his Madawaska office after the court hearing and that his editors asked him to write a story. It was published the next morning on the front of the newspaper's second section.

One of the 17 copies of the Daily News that go to Edmundston each day was bought at the courthouse newsstand by Paul Duffie, Gagnon's defense attorney. Duffie complained, and authorities prosecuted.

Banville appeared at his Canadian trial voluntarily, rather than force Canada to go through the State Department to have him extradited. If he had not, the Canadians could have issued a warrant for his arrest and picked him up the next time he crossed the river.

On Aug. 25, Canadian Provincial Court Judge James D. Harper found Banville guilty of violating the gag order because some copies of his newspaper had been sold in Canada, and fined him $200. The maximum penalty for a single violation is a $500 fine and six months in jail.

The Daily News has filed an appeal expected to be heard in Canada this fall. The appeal is based on the Canadian court's definition of "publish" and the question of whether a Canadian court has jurisdiction over a Maine company.

If anyone were to be held liable for publication of the article, the newspaper has complained, it should have been the company, not the reporter.

"The judge said he would not have found the newspaper guilty," Reynolds said. "If we didn't commit a crime, we don't see how the reporter could have. The reporter's not responsible for what we decide to print."

Banville is the first person prosecuted under the Canadian law, which took effect in 1969. A similar problem arose several months ago in Buffalo in a widely publicized case in which a Canadian woman, later acquitted of murder charges, dropped her baby into the water on the Canadian side of Niagara Falls. The judge in that case also banned news reports on the preliminary hearing.

Edward Cuddihy, assistant managing editor of the Buffalo Evening News, with a circulation of 10,000 in Canada, said the newspaper feared that Canada would seize its delivery trucks on that side of the border if the newspaper directly violated the order.

Instead, Evening News reporters sat through the hearing, then interviewed each witness outside the hearing room to reobtain their testimony. The newspaper published a front-page story and has not been challenged by the court.

Cuddihy said that editors had discussed simply covering the hearing and omitting the story in editions circulated in Canada, but that they then decided that would be impractical.

Editors are worried that if the issue resurfaces, witnesses may not be as cooperative. Cuddihy said that unless the law is clarified, a decision on whether to violate a judge's order would depend on the story's importance and whether the information could be obtained in any other way.