The star prosecution witness in a trial here of 16 black leaders on treason charges told the court last week that he had misunderstood his role in the case and made "fundamental mistakes" in his evidence that could have misled the court.

As Judge John Milne adjourned the case until today for prosecution lawyers to consider the implications of the witness' admission, legal experts said the case could collapse.

If it were thrown out, which could not happen until the prosecution closed its case, it would mean a further political embarrassment for South Africa over its handling of political dissidents. The prosecution's presentation could take several weeks.

Already the government of President Pieter W. Botha has been accused of using trials such as this as an extension of its detention system, tying up political opponents in long, complicated cases that put them out of action for months or even years.

Some of the 16 being tried here -- who include five of the "Durban Six" who sought refuge in a British Consulate last year -- were first detained under the security laws in August 1984. The others were detained in February.

When they were charged, the government issued a special order prohibiting the judge from granting them bail. When they challenged the order in the Appeal Court and won, bail of up to $5,500 was set, with stringent conditions preventing them from taking part in any political activities.

Even if the case against them were to collapse now, it would mean that 16 top black leaders, including two copresidents of the influential United Democratic Front, effectively will have been put out of action for the whole of this year. The front is perhaps the most strongly supported black political movement in South Africa.

The cross-examination of the prosecution's star witness, Isaak D. de Vries, brought the trial to crisis point. De Vries, 30, is a lecturer in politics at the Rand Afrikaans University in Johannesburg who is working on a PhD thesis on revolutionary strategy.

He has been presented as an expert on revolutions, and his testimony has laid the theoretical foundation for the treason charges against the 16 accused. It is considered vital to the prosecution's case.

This is the 20th case in which de Vries has testified for the state in cases against black dissidents. In most his testimony has gone unchallenged.

The essence of his testimony is that a "revolutionary alliance" exists between underground insurgent organizations such as the African National Congress, and legal organizations that share the same aims but can operate openly.

Citing Mao Tse-tung's maxim that the masses are to the guerrilla fighter what the sea is to the fish, de Vries contends that the role of the legitimate organizations is to mobilize and politicize the masses in support of the revolutionary underground.

De Vries' role in the prosecution case has been to establish a connection between the legitimate organizations, to which the accused belong, and the ANC and other underground members of the "revolutionary alliance."

He has sought to do this by pointing to a number of common "revolutionary symbols" -- colors, songs and rallying calls -- that have appeared at United Democratic Front and other meetings which the accused attended. This is where the videos and reports by police informers come in.

Under cross-examination by the defense lawyer, who is Indian, de Vries admitted that the Natal Indian Congress and Transvaal Indian Congress, to which many of the accused belong and which he initially described as committed to violent revolution, were in fact guided by the nonviolent philosophy of their founder, Mohandas K. Gandhi, who lived in South Africa before he founded modern India's independence movement.

After the defense attorney accused the young lecturer of being "a theoretician who has no factual background and no expertise to make statements or assessments," de Vries conceded that none of his opinions could be regarded as conclusive.

He said he had misunderstood his role in the case, believing he had only to point out the "revolutionary symbols" to the court and not draw conclusions from them. He admitted making "fundamental mistakes" in his evidence, along the lines of the characterizations of the Indian parties, that could have misled the court.

After the court had adjourned twice last week to assess the implications of de Vries' replies, the defense finally applied for a third adjournment on Thursday "with the ultimate object of curtailing these proceedings." The request was granted.

The case already had offered some embarrassing moments for the authorities. A security police officer, Maj. Harold Miles, revealed under cross-examination last week that police informers were paid according to the information they gave, getting more money the more valuable the information.

The major admitted that this could be an incentive to informers to exaggerate their reports, and that informers' reports often formed the basis for issuing restriction orders against dissidents.

Miles testified that he had sent an informer, "a student who was working for pocket money," to a political rally that some of the accused leaders addressed in Durban in August 1982.

Miles said he assessed the value of the student's information and paid him the equivalent of about $10.