The United States ordered South Africa's two top-ranking defense attaches here to leave the country as retaliation yesterday against the Prestoria government's expulsion of three U.S. military officials for alleged spying activities.

The move immediately heightened speculation that the deteriorating relations between Washington and Prime Minister Pieter W. Botha's white minority government could result in South Africa's withdrawal from the American-backed plan to grant independence to Namibia under United Nations supervision.

Washington's decision to expel Comodore Willem N. du Plessis, the defense and naval attache, and Col, Gert J. Coetzee, the air attache, came less than 24 hours after the South Africans accused the three Americans of taking aerial photographs of secret installations from the embassy's official plane.

State Department spokesman Hodding Carter refused to specify the reasons why the South Africans are being expelled. But the tone of Carter's remarks indicated that the U.S. government is offended both by the South African charges and Botha's action in going on television to announce the ouster of the Americans only minutes after notifying U.S. Ambassador William B. Edmondson.

An American offical, who asked not to be identified, said Botha's tactics have raised the question of whether his government, beset by a serious domestic scandal involving improper use of government funds, is trying to implicate the United States in an alleged spy plot as an excuse for pulling out of the Namibia agreement.

In reference to South African charges, Carter said, "I can assure you there won't be an apoloyy. We do not feel we have any reason to offer one."

Various official sources have confirmed that South African authorities discovered an aerial camera aboard the embassy plane, and there have been reports from South Africa that officials there also found allegedly unauthorized photos.

Carter refused to discuss the specifics of these reports or to deny out-right the spying allegations. But a U.S. officials, while also not denyihg that photos were taken from the plane, pointed out that U.S. spy satellites are able to obtain pictures of military installations and other restricted areas much more efficently than can be done with what he called "a relatively primitive camera."

The official also said that the itineraries of all flights made by the embassy plane had been carefully reported to South African authorities and that they were fully aware of how the plane was used. In fact, the official added, the plane has beenpiloted several times by South African military off* icer, and at least three high ranking South African government officials have been passengers on it.

In South Africa, Foreign Minister Roelof (Pik) Botha, who is not related to the prime minister, called the order for the two attaches to leave Washington within a week "a purely retaliatory action" and demanded anew that the United States apologize for its alleged spying.

Washington Post Correspondent Caryle Murphy reported from Johannesburg that the hard-line U.S. response came as something of a surprise in South African official circles. In the past, the Carter administration has tended to treat South Africa somewhat gingerly in order not to disturb the already shaky negotiations over a Namibia settlement.

Namibia, formerly known as South-west Africa, is a sparsely populated, but mineral-rich, territory whose predominantly black population has been ruled by South Africa under an old League of Nations mandate since the end of World War I.

For the past decade, it has been the scene of sporadic fighting between South African forces and the guerrillas of the Southwest africa People's Organization (SWAPO). South Africa, while agreeing that Namibia should become independent, is reluctant to see the new country come under the control of SWAPO, which Pretoria regards as communist-controlled.

Last July, following a long U.S. initiated diplomatic effort involving countries on three continents, both Pretoria and SWAPO agreed to an independence plan that would have the United Nations supervise elections for a Namibian government.

The plan was hailed widely as a triumph for U.S. diplomacy, and the administration is very anxious to see it carried through successfully to demonstrate that other southern Africa racial conflicts like the one in Rhodesia can be resolved peacefully.

However, progress toward that goal has been impeded by a number of problems, including heavy domestic opposition toward the plan within South Africa and the Botha government's vulnerability to revelations of an internal corruption and influence-buying scandal.

Efforts to resolve these problems reached a make-or-break stage last month when South Africa charged that the U.N. election plan would give SWAPO an unacceptable advantage. At present, Pretoria is studying some new proposals, worked out by U.S. and U.N. officials.

At the same time, though, South Africa has engaged in an escalating series of harsh attacks on U.S. motives and official, including Donald McHenry, the U.S. diplomat who is the principal architect of the Namibia plan.

Now, this campaign has been capped by the spying charges against the American military attaches. That's why a U.S. official, citing this chain of events, sugested yesterday that the Botha government, to delfect attention from its internal problems, "may be positioning itself for rejection of the Namibia agreement by casting things in terms of a spy plot involving major U.S. government figures."