South Africa's embattled white minority government accused the United States yesterday of mounting a "systematic program" of aerial spying through the U.S. Embassy in Pretoria and ordered the embassy's defense attache and two aides expelled.

Appearing on a television newscast last night without prior warning to the Carter administration, South African Prime Minister Pieter W. Botha voiced "shock and dismay at this reprehensible act" which, he said, he would have expected from the Soviet Union but "not from the leading Western country."

While the State Department and the embassy declined any detailed comment on the charges, U.S. officials here acknowledged privately that South African forces had seized aerial photographic equipment from the defense attache's small, twin-engine C12 aircraft last Friday.

Botha's dramatized verbal assault and his demand for an official apology sharply escalated his government's running diplomatic controlversy with the Carter administration, which has been trying to negotiate with South Africa for a peaceful settlement to the guerrilla war in Namibia (Southwest Africa).

Administration officials said that while U.S. negotiation efforts would continue, Botha's sudden attack and a demand for an official apology greatly reduced the chances for success in the talks, which reached a final make-or-break stage this month.

These officials also speculated that the espionage charges would distract domestic South African attention from the still spreading corruption and influence-buying scandal that threatens Botha's six-month-old government.

Botha's announcement came as Secretary of State Cyrus R. Vance, national security adviser Brzezinski and other senior Officials were meeting at the White House for a review of U.S. policy toward Rhodesia.The spying charges were discussed at the meeting, U.S. officials said, but not decisions were taken to respond to the accusations or expulsions, U.S. officials said.

Botha charged that an embassy passenger-carrying aircraft used by Ambassador William P. Edmonson had been secretly outfitted with an aerial survey camera and converted into "a spy plane" that had covered "vast areas of South Africa, including some of our most sensitive installations." He said South Africa had obtained some of the photographs.

Calling the case "disgraceful" and "an abuse of South African hospitality," Botha said that an unspecified number of U.S. Embassy personnel were ordered yesterday to leave South Africa within a week. The embassy airplane will also leave the country as soon as South African technicians have made sure it no longer contains any equipment that could be used on its flight out.

The State Department later identified the personnel asked to leave as Air Force Col. Alvin Crews, the defense attache, Maj. Bernd McConnel, assistant air attache, and Master Sgt. Horace Wyatt, crew chief of the embassy plane.

U.S. officials noted that the department's statement contained no hint of an apology or admission of wrongdoing as demanded by Botha at a press conference after the televised an announcement. The officials confirmed that the attaches had taken photographs in South Africa, but described their work as routine information gathering rather than espionage.

The defense attache's office at the embassy in South Africa has 14 staff members and, as is the case at many large U.S. embassies round the world, maintains an aircraft to transport embassy staffers and important U.S. visitors.

Comments by U.S. officials here suggested that the South Africans should have been aware for some time of the reconnaissance activity of the twin-engine Beechcraft, which was often left unattended on a South African military airfield outside Pretoria.

Expectations of a new confrontation with Botha's government had been building in Washington since a series of unexpected and harsh attacks last week by Foreign Minister Pik Botha on the Carter administration and particularly on U.S. diplomat Donald McHenry as an "enemy" of South Africa.

McHenry, who holds the rank of ambassador, has been the chief negotiator for the five Western countries that have been brought South Africa and the guerilla forces of the Southwest Africa Peoples' Organization to the brink of an agreement on United Nations-supervised elections for Namibia.

But U.S. officials were clearly stunned by the bitterness and public drama of yesterday's attack on U.S. diplomats, which comes after a special effort by the Carter administration to ease relations with the new government.

South Africa reacted sharply to the Carter adminstration's early stress on human rights and its commitment to majority rule in southern Africa, as enunciated at the United Nations by Ambassador Andrew Young and by Vice President Mondale in a stormy meeting in Vienna with then-prime minister John Vorster.

Vorster's National Party used the Carter administration's "hostility" to the rule of the country's 4.5 million whites over the disenfranchised 18 million blacks as a major campaign issue in 1976 and again in 1978. But when Vorster stepped down last September and Pieter Botha succeeded him, Vance went to Pretoria with a conditional invitation to the new prime minister to visit Washington once the Nambia negotiations were successfully concluded.

But relations worsened again in recent weeks as Botha's government found itself increasingly boxed in on the Nambia negotiations, which had to be completed this month in order for U.N-supervised elections to be held this year as agreed.

Botha's government has been weakened at home by disclosures that some of the ruling party's top figures were involved in slush funds intended to bribe politicians and journalists abroad.

From Cape Town, where Botha appeared on television, news agencies reported that sources close to the government said that disclosures about the plane's equipment came from Rhodesia. A Rhodesian black opinion weekly, The National Observer, reported last week that the Beechraft, equipped with a camera, had visited Rhodesia last month. CAPTION: Picture 1, WILLIAM B. EDMONDSON . . . use of his plane alleged.; Picture 2, PIETER BOTHA . . . expresses "shock and dismay."