The chief of the South African Defense Force, Gen. Constand Viljoen, acknowledged today that three white soldiers killed in Zimbabwe Aug. 18 were members of the South African Army, but he said they were there on an unauthorized mission.

His admission follows a rash of accusations by the leaders of countries bordering South Africa that it is sponsoring clandestine military operations in their countries to destabilize them. This is also the second time recently that the South African military has said its forces were acting on their own when they have been involved in military operations in other countries.

It did so in June when evidence at the trial of a group of mercenaries who tried to stage a coup in the Seychelles last November showed 14 of the men were South African soldiers and all had been armed by the Defense Force.

Today the opposition Progressive Federal Party's defense spokesman, Philip Myburgh, said the two events raised serious questions about the state of discipline in the military.

Zimbabwean authorities showed the bodies of the three soldiers to journalists last Sunday, later saying there was also a fourth. Prime Minister Robert Mugabe said they were members of a South African unit sent to destabilize Zimbabwe.

For more than a year Mugabe has been claiming South Africa is training large numbers of Zimbabwean dissidents to carry out operations inside his country.

Although South Africa denied all of Mugabe's earlier accusations, it did not respond immediately to this one. Viljoen announced Monday that he had ordered an investigation. In what he described today as a preliminary report on this investigation, Viljoen said the three men were part of a group of 17 former Zimbabweans recruited into the South African Army who had taken it upon themselves to cross the border to try to free some political dissidents they believed were being held in southeastern Zimbabwe.

Viljoen said the men were based in the northern regions of Transvaal Province, and at the time were patrolling a three-mile-wide strip that runs between the Venda homeland, which South Africa regards as independent, and the Limpopo River, which forms the border with Zimbabwe.

They were there to guard against guerrillas of the African National Congress infiltrating from Zimbabwe, but Viljoen said their emotions became aroused when they heard about the dissidents being held in their former country.

They decided to try to free them, and crossed the river Aug. 14, Viljoen said.

Four days later they encountered a Zimbabwean Army patrol. Viljoen said 14 of the group, all blacks, scattered and returned to their base in South Africa. The three whites who were leading the group stayed to engage the patrol and were killed.

Viljoen implied that one of them, Sgt. John W. Wessels, was wounded and later killed by the Zimbabweans.

Viljoen said he would take strong steps to deal with any similar events in the future. But he said he would take no action against the 14 black soldiers, because they had been misled by the three whites into believing the mission was officially authorized.

This was another echo of the Seychelles operation, in which the soldiers said they went on it believing it to be an officially authorized secret operation. They said they were served with official call-up papers.

Mugabe's claims that South Africa is trying to destabilize Zimbabwe have been echoed by other leaders in the region.

President Samora Machel of Mozambique has claimed repeatedly that South Africa is aiding dissidents of the Mozambique National Resistance Movement.

The Angolan government accuses South Africa of aiding the rebels of Jonas Savimbi's UNITA movement, which operates throughout large regions of southern Angola.

Lesotho, a tiny mountainous enclave completely surrounded by South Africa, claims South Africa is assisting rebels of the Lesotho Liberation Army, which is trying to topple Prime Minister Leabua Jonathan.

They have offered no explanation of why South Africa should try to destablize them, except to suggest that the white segregationist government does not want to see stable black-ruled states develop on its borders and expose the folly of its race policies.

South Africa consistently denies these claims, saying the African leaders use them to mask their own inability to maintain order.