An angry Reagan administration demanded yesterday that South Africa explain why two of its commandos were killed and another captured in northern Angola Tuesday night only 300 yards from an American-operated Gulf Oil Co. facility.

State Department spokesman Edward Djerejian said the U.S. reaction to the South African military foray was "deep displeasure," which has been made known to the Pretoria government.

Other officials expressed consternation about what appeared to be "an unfriendly act by a supposedly friendly South African government," especially since it could have sabotaged not only a U.S.-operated oil installation but also the shaky southern Africa peace initiative sponsored by the United States.

U.S. Ambassador Herman Nickel was instructed to lodge a strong protest in Pretoria and demand a full explanation, State Department sources said. South African Ambassador Bernardus G. Fourie was summoned to the State Department for the second straight day to hear similar representations from senior officials, the sources said.

The apparent South African threat to a U.S. installation comes as sentiment in Congress is becoming increasingly hostile to South Africa and to the administration's "constructive engagement" policy of working closely with the Pretoria government to seek regional peace and amelioration of the apartheid system.

As South Africa sent a message to Angola requesting an urgent meeting to arrange return of the bodies of two commandos reported killed and a third commando captured, Washington Post special correspondent Allister Sparks reported strong criticism of the military action across a broad spectrum of South African political opinion.

The rightist Conservative Party, hawkish on defense, accused the government of bungling and of giving the South African Defense Force a bad name. Liberal spokesmen, meanwhile, questioned the government's claim that its troops were seeking information about guerrilla bases, noting that the oil-processing enclave of Cabinda in the far north is an improbable place for such bases.

In the message to the Angolan government, released in Pretoria, South African Foreign Minister Roelof F. (Pik) Botha justified the incursion on grounds that "South African security forces have felt it necessary to gather intelligence on the activities of the ANC African National Congress and SWAPO South-West Africa People's Organization terrorists in Angola and to take appropriate counteraction."

An Angolan Defense Ministry communique called the South African intruders "saboteurs" rather than intelligence-gatherers, and said they were caught "trying to destroy" the oil installation jointly owned by the Angolan government and Gulf Oil. The communique said the South Africans were armed with land mines, incendiary shells and boxes of powerful explosives as well as personal weapons.

Reports reaching the State Department said the South African commandos were caught by Angolan military forces only about 300 yards outside the Gulf Oil compound, where several dozen Americans along with Europeans work and are housed. Some local employes of the oil installation are reported to have heard the gun battle, which took place late Tuesday afternoon Angola time.

The Cabinda facility produces about 170,000 barrels a day of crude oil from offshore valued at more than $4 million. The proceeds, divided almost equally between Angola and Gulf, are the financial mainstay of the Marxist government in Luanda.

The State Department spokesman said the U.S. government views "most seriously any effort from any source which would have the effect of placing in danger Gulf Oil employes or property in Cabinda." Djerejian said the United States had "no knowledge of the possibility of any South African military operation in Cabinda" until the announcements after the clash.

"The United States deplores the presence of South African intelligence operations inside Angola as contrary to the aims of its diplomacy and the achievement of peace in the region," the State Department spokesman said. Among other things, he said, the operations are contrary to the Lusaka accords, to which the United States is a party, calling for South African withdrawal from Angola.