Terrorist attacks on the Rome and Vienna airports last December and the installation of Soviet-built antiaircraft missiles in Libya before that provided the catalyst for President Reagan to launch a military challenge to Libyan leader Muammar Qaddafi, administration officials said yesterday.

After these incidents and mounting frustration over intelligence reports showing that Qaddafi was continuing to send terrorists to European capitals, Reagan and his senior advisers decided it was time "for a real get-tough attitude" toward Qaddafi, a senior administration official said.

The detailed planning for yesterday's action began shortly after the Dec. 27 airport attacks and was carried out with the expectation that Libya would strike at U.S. forces if they crossed Qaddafi's "line of death" in the Gulf of Sidra. Officials said the SA5A missile site at Surt had been targeted in advance for retaliation if Libya fired on American planes.

Although the White House claimed yesterday that the purpose of the naval exercise was solely to demonstrate freedom of navigation in an international waterway, officials said privately that the exercise was planned with a realization that it might provide a military confrontation with Qaddafi and a chance to underscore Reagan's determination to deal firmly with international terrorism.

In approving the exercise March 14, officials said Reagan, after a discussion with senior officials, decided to give commanders on the scene specific authority in advance to destroy the missile site at Surt. Officials also said they discussed the potential for attacks by Libyan patrol boats.

The last U.S. naval exercise in the region involved only two aircraft carriers, and officials said yesterday the latest exercise was held up until a third carrier was in position. Pentagon officials, who in the past have been extremely cautious about the use of military force in such situations, approved the current plan but insisted that "everything be in place" before the exercise began.

A senior administration official said planning for the exercise, involving the Central Intelligence Agency, State Department, Defense Department and White House, had been under way for several months. While it was coordinated by national security affairs adviser John M. Poindexter, the plan was described as an administration-wide effort with the primary impetus coming from Secretary of State George P. Shultz.

But officials said Defense Secretary Caspar W. Weinberger also approved of the plan, and, accompanied by the chairman of the Joint Chiefs of Staff, Adm. William J. Crowe Jr., visited the Oval Office yesterday to discuss it with the president. The subject also came up briefly at Reagan's weekly issues lunch as Poindexter frequently left the room to get updates for the president on the movement of U.S. forces in the gulf.

The Reagan administration has long been in conflict with the Qaddafi regime, which it regards as a destabilizing force in the Middle East and a principal source of international terrorism. In August 1981, two U.S. F14 fighters shot down two Soviet-built Libyan fighters after a brief dogfight over the gulf.

U.S. concern over Libyan intentions was heightened last December by the disclosure that the SA5A missiles, which have a range of about 150 miles, were being installed. Officials said then that the missiles had the capability of knocking out U.S. reconnaissance aircraft, including sophisticated AWACS planes, though not high-performance fighters.

Denouncing the Libyan installation of the missiles on Dec. 20, State Department spokesman Charles Redman said, "This clearly exceeds any legitimate security requirements the Libyans have. This is a significant and dangerous escalation in the Soviet-Libyan arms relationship."

The United States protested to Moscow but was rebuffed in a reply that Redman said "did not address our concerns."

A week later the terrorist attacks took place at the Rome and Vienna airports. They angered U.S. officials, who at first encouraged an Israeli retaliation against Libyan targets and then reversed themselves after conflicts developed within the administration about the appropriate response.

Instead of a military response, the administration with considerable fanfare decided on economic sanctions and ordered Americans, primarily businessmen and oil company workers, to leave. Reagan announced these steps Jan. 7 at a news conference in which he said "Qaddafi deserves to be treated as a pariah in the world community."

Reagan warned that if the sanctions did not work, he was prepared to go further to curb Qaddafi.

"If these steps do not end Qaddafi's terrorism, I promise you that further steps will be taken," the president said.

Most U.S. allies have refused to join the economic sanctions or have applied them only partially, and U.S. officials acknowledge that they have been less successful than they hoped.

Ever since Reagan vowed soon after his inauguration that terrorists faced "swift and effective" retribution from the United States, his Cabinet and top military advisers have been in conflict about how to accomplish this goal. Reagan's senior advisers differed over how to respond to several terrorist attacks, including the suicide bombing of U.S. Marine headquarters in Beirut in 1983, the murder of four Marines in El Salvador last year and the hijacking of TWA Flight 847.

Shultz has consistently pushed for military action, even at the risk of harming civilians. Weinberger and the Joint Chiefs have argued that any use of military force must be carefully weighed against public opinion and possible reaction against such actions. After the TWA hijacking, Reagan ruled out retaliation unless it was directed at specific perpetrators, and said that the killing of civilians would be "an act of terrorism itself."

The only previous time that Reagan ordered American military forces into action against terrorists was the interception of an Egyptian plane carrying the hijackers of the Achille Lauro cruise ship last fall. That was a quick and limited operation carried out without shots being fired, and one that was managed largely within the White House.

In contrast, yesterday's engagement with Libya leaves open the prospect of a wider military confrontation. A senior official said the U.S. naval exercise will continue for the next six or seven days and the United States does not intend to retreat.

"If they continue to provoke us, we will continue to respond," he said.

Expressing the widely held administration view of the Libyan leader, another administration official said, "You don't know what you're dealing with. This man is truly erratic. No one knows what he's going to do."