The trial of Jewish activist Viktor Brailovsky on charges of anti-Soviet defamation opened here today with police barring all but his family and state-approved spectators from the courtroom.

Among those barred were diplomats from the United States, Canada and Great Britain.

A cybernetics expert denied permission to emigrate to Israel for more than eight years, Brailovsky was arrested last November at the end of a long campaign of harassment by authorities seeking to suppress a scientific seminar he helped organize and a samizdat, or self-published journal, of Jewish thought that he helped edit.

According to his wife, Irina, who with their son, Leonid, was allowed to attend today's session, of the four witnesses called by the prosecutor, one refused to testify, another said he could not remember evidence shown him, and a third said he had never read the journal "Jews in the U.S.S.R."

Mrs. Brailovsky said the fourth state witness testified that he had once attended a scientific seminar at the family's apartment that had dealt only with mathematics and physics. But the prosecutor alleged that the seminar had discussed topics that defamed the state.

She said her husband, a slight, bearded 45-year-old who has been unemployed since applying to emigrate, looked "terrible," with a pale, puffed-up face. It was the first time she had seen him since his arrest.

Interior Ministry officials have thwarted her repeated efforts to send medicine to him for a chronic kidney ailment, she said.

Brailovsky's arrest just before the Madrid Conference on European Security began has brought continued protests from Western scientists.

The political charge of "circulation of fabrications known to be false which defame the Soviet state and social system" carries a maximum sentence of three years' imprisonment.

Mrs. Brailovsky said the state alleges that "Jews in the U.S.S.R.," one of the oldest self-published journals in the country, defames the state by asserting the Soviet Union discriminates against Jews, encourages Russian chauvinism and oppresses culture. But she said the prosecutor made no direct connection between what was alleged to be in the journal and her husband's editorship. Brailovsky was the last of a succession of editors of the journal. The others have been allowed to emigrate over the years.