British Foreign Secretary Geoffrey Howe left here empty-handed tonight at the end of a weeklong mission for the European Community that South African President P.W. Botha assailed as interference in South Africa's affairs and an attempt to force the country to commit "national suicide."

Howe was reluctant to characterize the mission as a failure, saying at a news conference before leaving that he thought he had made a "contribution" by impressing on Pretoria the need to release imprisoned black leader Nelson Mandela, lift the ban on Mandela's African National Congress and begin negotiating with the outlawed group.

But Botha, indicating that a meeting between the two men earlier in the day -- the second of Howe's visit -- had been abrasive, accused Howe of not being interested in hearing about positive political developments here and said he was determined not to be pressured by Howe.

"He came to South Africa mainly to bring pressure to bear on us to unconditionally release Mandela and to unban the ANC," Botha said.

The president made it clear that he was not prepared to do this as long as the ANC was "under communist control" and remained committed to forcing political change by guerrilla tactics.

"I can never commit suicide by accepting threats and prescriptions from outside forces and handing South Africa over to communist forces in disguise," he said.

Botha said he hoped "this hysterical outcry" by western countries against South Africa, a white-minority ruled country, would soon pass. But, he warned, if it did not, and sanctions were applied, South Africa would resist. "If we are forced until our backs are against the wall, we will have no alternative but to stand up in self-respect and say to the world, 'You won't force South Africans to commit national suicide.' "

Howe said he hoped the South African government had understood his message that the key to the country's future lay in the black leaders' hands and that they needed to make "a leap of faith" by releasing jailed black leaders and beginning a dialogue with them.

But Botha showed no sign of acknowledging this when he stressed his determination to press ahead with his own limited reform program by negotiating with blacks other than Mandela and the ANC through a newly established national council.

Although only blacks who are already involved in administering the apartheid system are known to be willing to participate, Botha said he was optimistic about the council's prospects of success.

"There are enough authentic and representative leaders in this country for us to iron out our differences," the president said.

"Leave South Africa to the South Africans," Botha admonished.

The notion that black majority rule would mean "national suicide" for the Afrikaners has long been a stock phrase in white politics here. It reflects the Calvinist Afrikaners' deep-rooted belief that they have a God-given right to exist as a distinctive volk, or nation, and that this would be imperiled if they became a minority group in a country run by others.

Howe's mission was undertaken as many western countries approach a time of decision-making on South Africa. Next week a committee of Commonwealth heads of state will meet to decide whether the 49-nation alliance of Britain and many of its former colonies should impose sanctions on Pretoria following the failure of a seven-member Commonwealth mission to find meaningful evidence that apartheid was being abandoned. The 12-nation European Community is scheduled to meet in September to decide whether to impose sanctions. Meanwhile, the U.S. Senate is considering sanctions legislation as the Sept. 9 date approaches for a review of President Reagan's executive order imposing limited sanctions.

Howe's mission was ostensibly to advise the EC for its September meeting, but black nationalists here, including Mandela, saw it as a maneuver by British Prime Minister Margaret Thatcher to sidestep pressure from her European and Commonwealth partners to apply sanctions. Because of this they refused to see Howe.

The only black person of any political consequence Howe saw was the influential Zulu homeland leader, Chief Gatsha Buthelezi, who is opposed to sanctions.

Howe said it had become clear to him during his visit that the political change already begun here had to reach the fundamentals of political rights. If this was to happen peacefully, it would have to be based on dialogue between the government and freely chosen black leaders. That, in turn, would not be possible until Mandela and other imprisoned leaders were freed and the ANC legalized in exchange for a matching ANC commitment to end its campaign of violence and start talking.

"In short, each side needs to make an offer that the other cannot refuse," Howe said.

Buthelezi made a new proposal for mediation to Howe, suggesting that a distinguished foreigner be appointed to shuttle between an internal group of South Africans made up of representatives of government, local black leaders and major industrialists, and an external group that would include representatives of the ANC. The mediator's task would be to try to bring the two groups closer together.

There has been no reaction to the suggestion from Howe or from South African political sources.

In another development, a ruling by the Supreme Court of Transvaal province invalidated the gagging of 119 antiapartheid organizations and the banning of many black activist meetings under the emergency regulations introduced six weeks ago.

It was the latest in a series of judgments striking down important aspects of the emergency regulations.

The judgments are particularly bold because they have overridden a clause in the emergency proclamation prohibiting legal challenges. That clause, too, has been declared invalid. This highlights one of the most contradictory features of South Africa, which continues to have an independent judiciary alongside many features of a police state.

Yesterday's judgment followed an action brought by the United Democratic Front, the main black activist movement here, which has been hardest hit by the government's crackdown. The judges ruled that although the president could delegate the power to issue prohibition orders to the commissioner of police, the commissioner could not in turn delegate this power to officers under him.

This invalidated hundreds of orders that have been issued by regional police chiefs prohibiting meetings and gagging organizations.

Legal specialists point out that the prohibitions can be reinstated by the commissioner reproclaiming them, but it is thought unlikely to be done with all of them.