JOHANNESBURG, NOV. 17 -- A South African provincial Supreme Court today rejected an appeal to release from prison a widely known black newspaper editor, Zwelakhe Sisulu, who has been held without charge for almost a year.

Sisulu, editor of the New Nation weekly newspaper, was ordered returned to prison and to pay court costs for an appeal based on a claim that his detention was illegal.

About 50 South African and foreign journalists were in the courtroom today to hear Sisulu's attorneys argue that the June 12 state of emergency, under which Sisulu was arrested, was illegal and that his detention was unlawful.

Supreme Court Justice Lawrence Harms ruled that Sisulu was a "danger to public order and public safety" because of his support for the National Education Crisis Committee, an organization opposing the system of strict segregation known as apartheid.

Sisulu, a former Nieman fellow at Harvard University, is the son of Walter Sisulu, who for the past 23 years has been serving a life sentence in Cape Town's Pollsmoor Prison along with African National Congress leader Nelson Mandela.

His mother is Albertina Sisulu, a national president of the United Democratic front, a coalition of 700 antiapartheid organizations.

With the release last month of former African National Congress national chairman Goven Mbeki, 77, from Cape Town's Robben Island Prison, there have been signs that the South African government may soon release Walter Sisulu and possibly Mandela in an effort to open a dialogue with the ANC.

In his appeal for release, Zwelakhe Sisulu said he was still "prepared to use his position in the community to ensure peaceful and just conditions in the country."

Sisulu was first detained June 26, 1986 and, after being released for a few weeks, was detained again Dec. 12.

The New Nation is a weekly newspaper published by the Southern Africa Catholic Bishops Conference.

At an international journalists' conference here in October, Home Affairs Minister Stoffel Botha emphatically denied that Sisulu was being detained for his political views or because of his journalistic activities.

Botha said he did not know why Sisulu was being held, but other government officials have said it was because of speeches he has made before the National Education Crisis Committee urging black schoolchildren to oppose the white minority government.

Sisulu has denied government charges that he is an officer of the education committee.

Adriaan Vlok, the minister of law and order, said in an interview last month that Sisulu was being detained for "security reasons" and not because of his activities as a journalist. Vlok declined to specify what security violations Sisulu was being held for.

Meanwhile, the black newspaper, The Sowetan, has been given notice by Home Affairs Minister Botha that it is in danger of being closed for promoting the outlawed African National Congress.

Botha said in a letter to The Sowetan, which is published by the white-owned Argus Newspaper chain, that he had read six issues of the newspaper and had found "prima facie" evidence that it had "tended to promote the public image or esteem" of the ANC.

Botha's written warning appeared to be the first step in a process that could lead to the closing of the newspaper.