Two South African newspaper editors and a reporter are being tried in secret here on charges that they exposed South African intelligence service activities after a bungled coup attempt by a group of mercenaries in the Seychelles Islands, off East Africa, just over a year ago.

The editors are Rex Gibson of the Johannesburg Rand Daily Mail and Tertius Myburgh of its sister paper, The Sunday Times. The reporter is Eugene Hugo, who wrote for both papers from the Seychelles after the coup attempt.

The secret trial takes place against a background of rising international accusations that South Africa is following a policy of "destabilization" against neighboring black-ruled countries, and that the coup attempt in the Seychelle Islands was part of this.

The three journalists face the possibility of imprisonment for up to 10 years, or a fine of $8,500, or both, under a new press law which prohibits the publication of any information relating to a security matter which might prejudice the country's interests.

They are accused of doing this by publishing a report, written by Hugo, that named two South African intelligence agents supposedly sent to the Seychelles to assess the damage done to South Africa's intelligence network there after the coup attempt failed. The case is being heard by the senior regional magistrate of Transvaal Province, J.A. van Dam, who is expected to deliver his verdict Friday.

When the trial began yesterday van Dam ruled that it be heard in secret under a clause in the new law, called The Protection of Information Act.

Reporters and members of the public were ordered out, and the large wooden doors of the courthouse were firmly closed.

Even details such as the pleas entered by the journalists and the names of witnesses called may not be revealed.

The lawyer for the journalists, Ernest M. Wentzel, said today he was not even allowed to say what stage the trial had reached.

The attempt to overthrow the socialist government of President Albert Rene was launched from South Africa by a group of mercenaries hired here, which included 14 commandos of the Army Reserve. It was led by the Irish-South African mercenary colonel of Congo war fame, Michael (Mad Mike) Hoare.

At his trial last year for hijacking an airliner to escape back to South Africa when the coup attempt failed, Hoare said the operation was carried out with the connivance of South African military and intelligence services, but the judge said there was no proof that the operation had been officially authorized.