Soviet-made SA5 long-range antiaircraft missiles are being installed in at least two locations in Libya, and the United States has protested the action to Moscow and been rebuffed, Reagan administration officials said yesterday.

"This clearly exceeds any legitimate security requirements the Libyans have," said State Department spokesman Charles Redman. "This is a significant and dangerous escalation in the Soviet-Libyan arms relationship.

"We have made clear to Moscow our concern about this escalation and Soviet support for an irresponsible and erratic regime," Redman said. Asked about the Soviet reply, Redman said, "The Soviet response did not address our concerns."

Deployment of the missiles -- which U.S. officials expect will be manned by Soviet troops -- increases the already-high tensions between the United States and Libya, which has been a major preoccupation of the Reagan administration.

Administration sources disclosed that precautionary military planning was initiated last summer to counter Libya if it attacked a neighboring North African state or was shown to be responsible for a major terrorist incident.

U.S. officials prepared military plans that an official said were "defensive" and "of a precautionary nature." One source said the Central Intelligence Agency analyzed military options and studied in detail the potential military targets in Libya. He said that a high-level U.S. emissary was sent to the Middle East to talk with friendly nations about coordinating possible military options.

But a Pentagon analysis of possible direct U.S. military action against Libya painted a bleak chance of success and effectively argued against it, sources said. The Pentagon said that, in the worst case, an American military operation could eventually require a commitment of six divisions or 90,000 men.

A senior administration official said this week that the military plan was never completed or submitted to the White House for action. The plan was never approved by Reagan, and subsequent discussions with him about the proposal were limited because the planning occurred while he was recuperating from his colon cancer surgery, officials said.

Last month The Washington Post reported that Reagan had authorized the CIA to undermine Qaddafi's regime covertly, another manifestation of the administration's concern with Libya.

The SA5, though a relatively old and slow-flying ground-to-air missile, can hit targets 95,000 feet in the air and has a range of about 150 miles, which could enable it to knock down U.S. reconnaissance aircraft including sophisticated AWACS planes, but not high-performance fighters. The Soviets have previously given them other antiaircraft missiles in the SA series, but none with a range of over 40 miles, and none that provided the same capability to knock down U.S. reconnaissance planes flying over the Gulf of Sidra, which Libya considers its territorial waters. The United States regards the gulf as international waters.

In August 1981 two U.S. F14 fighters shot down two Soviet-built Libyan fighters after a brief dogfight over the gulf. The SA5 missiles have the theoretical capability to down fighter planes but are considered by military sources to be of relatively limited value when employed against such high-performance aircraft as the F14.

Anthony H. Cordesman, a Middle East arms specialist, said of the Soviet delivery of the SA5 missiles: "It's a low-cost way of bugging the hell out of Israel, Egypt and the United States. SA5 is a museum piece, a symbolic gesture. It moves very slowly and is jammable."

However, administration officials consider the installation of the missiles symbolically important because they bolster Qaddafi's challenge to U.S. interests in the region and escalate the Soviet commitment to his government.

Increasingly, U.S. analysts have identified Libya as the persistent source of what Secretary of State George P. Shultz and other officials have called "state-supported terrorism." This concern became extreme in mid-July, after an American was killed and 39 others held hostage for 17 days on hijacked Trans World Airlines Flight 847.

While Reagan and his advisers won considerable public support and praise for their handling of this incident, the appraisal within the government was more pessimistic. According to administration sources, the issue was discussed with the president in mid-July shortly before he underwent surgery for colon cancer. Reagan reportedly agreed with the views of CIA Director William J. Casey; Robert C. McFarlane, then national security affairs adviser, and other top aides that the administration had been fortunate in the outcome of the crisis and needed contingency plans to deal with possible future incidents. That led to the drawing up of plans for possible military action against Qaddafi.

Qaddafi visited Moscow Oct. 10-12 where he sought but failed to obtain a treaty of friendship and cooperation with the Soviet Union, assistance in building a nuclear reactor and a more favorable treatment for repayment of his $4 billion to $5 billion debt to the Soviets.

According to U.S. diplomatic sources and other Middle East specialists, the visit did not go well. One U.S. official said that discussions between Qaddafi and his Soviet hosts were "acrimonious" and that the Soviets gave the Libyan leader "a dressing-down" for his support of terrorist activities.

Despite the difficulties on this visit, one source said Qaddafi left with a general statement of support from Soviet leader Mikhail Gorbachev. It is not clear whether the deal for acquisition of the SA5 missiles, which Libya has been seeking since 1983, was struck at this Moscow meeting.

Military sources said the missiles began arriving at the Libyan port of Misrata in November. A U.S. arms specialist said that one of the sites at which the missiles are being deployed is the old U.S. Wheelus Air Base on the outskirts of Tripoli, which Qaddafi took over after coming to power in 1969. A second site was said to be near Benghazi on the Mediterranean coast in eastern Libya. A third site, which is not fully confirmed, was said to be at the oasis of Kufra in southeast Libya near the Egyptian border.

American AWACS radar planes have operated in the past from inside Egypt to observe activities in Kufra, Libya's main military staging area near the borders of Chad, Sudan and Egypt.

Sources said that the first SA5 missiles could become operational within five months and would be manned by Soviet crews. Reports differed on the number of missiles that would be deployed, ranging from 36 to 54.

ABC News and columnist Jack Anderson both reported in November that Libya was receiving Soviet SA5s, but yesterday's statement by the State Department's Redman was the first official confirmation.

State-supported terrorism has been a persistent theme and major frustration for the Reagan administration since 241 U.S. servicemen were killed by a suicide truck bomb at U.S. Marine headquarters in Beirut on Oct. 23, 1983. While the administration has often vowed swift retribution against terrorism, it has struggled without success to find effective ways in which to retaliate.

From the U.S. viewpoint, Libya has been the most flagrant supporter of anti-U.S. terrorist groups and their activities. The Arab world's foremost terrorist, Abu Nidal, is believed to be living in Libya and to be coordinating his anti-American activities with the Libyan government. U.S. sources have said there was some reason to believe Abu Nidal was responsible for the hijacking of an Egyptian jetliner to Malta last month.

The Soviet Union has already sold to Libya about $15 billion in arms, such as 2,800 tanks and 450 combat aircraft, including MiG23 and MiG25 fighters. But much of the Soviet weaponry remains unused and in storage, making Libya in effect a Soviet arms depot in the Middle East and northern Africa.

The only other country in the region that the Soviets have armed with SA5 missiles is Syria, which has used them against Israeli aircraft and has deployed them along its coast as part of its air defense system.