Sen. Edward M. Kennedy (D-Mass.) canceled the final rally of his South African tour today after security aides decided that militant black demonstrations against him in Johannesburg's black township of Soweto could turn violent.

Two factions -- one chanting "yes" and the other "no" to letting Kennedy speak -- marched toward each other in Soweto's Regina Mundi Cathedral, where the speech was to take place, and met head-on in front of the altar.

Anglican Bishop Desmond Tutu, a Nobel Peace Prize winner and outspoken foe of South Africa's system of racial segregation known as apartheid, finally gained control over the nearly riotous crowd of 4,000. By calling for a vote, he established that the overwhelming majority wanted Kennedy to appear. But he, too, decided in favor of calling off the meeting to avoid possible intervention by South African police, who were standing by with three armored vehicles.

The stormy confrontation in the Roman Catholic cathedral, a traditional Soweto meeting place, was the climax to a range of powerful emotional reactions that Kennedy's eight-day tour has stirred among white and black South Africans.

As a parting shot, Foreign Minister R.F. (Pik) Botha wrote Kennedy a letter today saying: "You did not come here to establish the facts. Your motive was to use your visit as a forum to obtain publicity for a set of preconceived value judgments."

Whites, including many liberals, have reacted to Kennedy's public spotlighting of some widely denounced features of the country's apartheid system with an indignation that one local columnist described as "a righteous frenzy."

Blacks have been divided between those who were cool toward Kennedy when he arrived but warmed as they heard him denounce apartheid in strong terms, and a vociferous minority of radicals who regard him as a representative of "capitalist imperialism" and have demonstrated against him wherever he has gone.

The strong anti-American and anticapitalist themes of these demonstrations have alarmed members of the Kennedy party, who say the Reagan administration's policy of "constructive engagement" with Pretoria is aggravating a trend in the black community that associates capitalism with racial oppression.

In an interview before leaving South Africa for Lusaka, Zambia, this afternoon, Kennedy said that as soon as he returned to Washington he would begin lobbying for legislation "to dissociate the United States from that concept" of constructive engagement.

"I have an awareness of how that can best be done," Kennedy said. "I'm interested in effective action and will have a series of proposals to offer."

Organizations adhering to the "black consciousness" philosophy of racial self-assertion propounded by the late black leader Steve Biko in the mid-1970s, but which have identified increasingly with socialism over the past two years, demonstrated against Kennedy from the moment he landed at Johannesburg airport Jan. 5. The black consciousness movement has its main strength in Soweto.

Tutu, in a farewell speech to Kennedy at the Johannesburg airport, referred to the white-minority administration and its police force collectively as "the system." He said: "We did not want to do anything that would give the system an excuse, for the system was waiting eagerly for an excuse to step in as if they were restoring law and order. We did not want to give them that excuse, and we did not want you to be used by them in that way."

Kennedy's tour has contrasted sharply with one made by his brother Robert in 1966.

Robert Kennedy was given a rapturous welcome by both blacks and white liberals wherever he went. Although the white-minority government disliked his idealistic, antiracist theme, he did not publicly attack the government and so drew little return fire.

But while that visit was aimed at encouraging the forces of liberalism, which seemed at the time to offer the main hope of peaceful reform in South Africa, Edward Kennedy has identified himself with the black nationalist struggle against apartheid and made a point of highlighting the most stark features of the segregationist system.

He visited a shabby hostel for black migrant workers in Soweto, which he described as "one of the most depressing and despairing" experiences of his life. He visited a black farming community and the Crossroads squatter community in Cape Town, both threatened with forced removal, and described the policy of uprooting people as "immoral and inhumane."

He walked through the cemetery of a resettlement camp and then questioned a white official in front of the television cameras about health facilities and the infant mortality rate.

He made a pilgrimage to see Winnie Mandela, who is under a banning order because of antiapartheid efforts. She is the wife of the imprisoned leader of the underground African National Congress, Nelson Mandela. When the government refused Kennedy permission to visit Nelson Mandela, he went to the gates of the prison where the black leader has been incarcerated for 22 years and made a statement expressing his admiration for him.

All this proved too much for the sensitivities of most white South Africans.

The correspondence columns of newspapers have been flooded with vituperative letters from white readers. One said: "I wish someone would explain to Sen. Kennedy the meaning of the word voetsak Afrikaans slang for 'shove off' because that is what he deserves."

The semiofficial broadcasting service spoke of the "overwhelming artificiality" of the Kennedy tour and its "blatant engineering of every situation to achieve maximum media impact."

Even some of the English newspapers that normally oppose the government joined the storm of outrage against him. They reminded their readers of Chappaquiddick and reports that Kennedy cheated in a Spanish examination when he was at Harvard.

Describing Kennedy as a "dilettante politician and morally inept," the country's leading financial publication, the Financial Mail, said: "Nothing about him suggests that he is able to, or in a position to, pass moral judgments on this part of the world."

Gerritt Viljoen, the minister in charge of black affairs, said after a private meeting with Kennedy Thursday, at which they discussed infant mortality and forced removals, "I have strong feelings about Sen. Kennedy's visit, but I would prefer not to divulge them."

Constitutional Affairs Minister Chris Heunis said no one had the right to pass moral judgments on other countries the way the senator was doing when there were injustices in his own country.

In his speeches, Kennedy repeatedly stressed that white South Africans should not fool themselves into thinking that "constructive engagement" signaled a tolerance of apartheid in the United States.

"Only a few extremists in my country still defend the government of South Africa. Patience is running out across the political spectrum," he said. "The people are marching ahead of government, and they will not permit it to turn back to business as usual with a regime of repression."