The Reagan administration sought today to prevent Israel from bombing neighboring countries or Palestine Liberation Organization camps in retaliation for Friday's terrorist attacks at airports in Rome and Vienna that left 18 dead and 113 injured.

"Every effort is being made to temper the expected Israeli response," a senior White House official said. "Exercise of any of the military options open to Israel would certainly inhibit the peace process in the Middle East."

Administration officials said that Reagan had sent a personal message to Prime Minister Shimon Peres deploring the terrorist attacks, which were directed at the facilities of El Al, the Israeli national airline. But these officials said this message and other diplomatic communications were aimed at preventing the kind of response made by Israel early in October when it bombed PLO headquarters in Tunisia in response to the killing of three Israelis on a yacht in Cyprus harbor.

The Israeli attack killed a number of civilians and drew a sharp protest throughout the Arab world.

A senior official said this sort of Israeli reaction would be "inapproriate" in the present situation since the terrorist attacks in Rome and Vienna appeared to be the work of a breakaway PLO faction known as the Abu Nidal group. PLO spokesmen have condemned the airport attacks.

Reagan's carefully worded message to Peres made the point indirectly and was designed, sources said, to show sensitivity to Israeli outrage about the airport attacks.

In a letter that began with the words "Dear Shimon," the president said he was "shocked and saddened" by the "vicious murders and wounding of innocent civilians," which he called "another example of the evil of terrorism that we must all work to eliminate."

"Such acts must be condemned and their perpetrators brought to justice," Reagan wrote.

"Further, we must not allow terrorists to deter us from pursuing our larger goal of a lasting peace," the message continued. "I extend to you and the Israeli people my most sincere condolences for the Israeli victims of this atrocity."

An official said that diplomatic messages had been sent to other governments urging them to "lean heavily on Israel" to prevent retaliation that would escalate the tense situation in the Middle East. Even before the terrorist incidents, U.S. officials were fearful of an Israeli military reaction against Syria, which reportedly has deployed surface-to-air missiles in Lebanon's Bekaa Valley and along the Syrian-Lebanese frontier.

The emphasis at the State Department seemed different, at least in tone, than the predominant calls for restraint being issued by the California White House, although officials there also were concerned about the overall military and diplomatic situation in the Middle East.

"Those who commit terrorism and support it must pay a price for it," said a senior State Department official. He declined to be specific about how and where such a price could be extracted but left the impression that U.S. action against those responsible for the deaths of at least five Americans has by no means been ruled out.

Nonetheless, there was concern at the State Department as well as the White House that Peres could be under domestic pressure to take actions that are only vaguely connected, if at all, with the Rome and Vienna attacks, but that would have serious, region-wide consequences.

In this category would be attacks on PLO headquarters installations in Jordan and attacks on fixed Syrian antiaircraft missiles on Syrian territory near the Lebanese border or mobile Syrian antiaircraft missiles that were sent back to Lebanon last weekend after previously being removed at U.S. and Israeli behest.

U.S. concern about potential Israeli actions today contrasted sharply with White House reaction after the bombing of the PLO headquarters in Tunisia. White House spokesman Larry Speakes originally issued a statement expressing understanding of this Israeli action, but subsequently modified his remarks and criticized the bombing at the insistence of Secretary of State George P. Shultz.

With the White House all but shut down for the holidays, Speakes held an impromptu briefing of reporters in his office at the Century Plaza Hotel here today and told them that messages had been sent to a number of governments in the Middle East urging them to "show restraint" in the wake of the terrorist attacks in Rome and Vienna.

"We're concerned with an escalation of violence on either side," Speakes said.

Officials here and in Washington said the administration also was concerned that any Israeli action would doom the frail peace process in the Middle East, unless limited to direct action against the perpetrators of the incident, as Reagan suggested in his message to Peres.

U.S. officials said that the United States would observe this standard, which has been enunciated frequently by Reagan in the past, in considering any retaliation for the killing of Americans in the terrorist bombings. A senior official conceded that this would make retaliation "very difficult" because of the shadowy nature of the Abu Nidal group.

As of this afternoon, State Department officials said information reaching Washington supported an increasingly clear conclusion, based on circumstantial evidence, that the attacks were perpetrated by terrorists associated with Abu Nidal, one of the most radical opponents of Arab peace with Israel and of Palestine Liberation Organization Chairman Yasser Arafat.

Among the circumstantial evidence of Abu Nidal's involvement, officials said, were reports that a note pinned to the clothing of one of the terrorists in Rome said, "Arafat is a traitor." Also fitting this pattern, officials said, were reports that Moroccan passports used by some of the terrorists in Friday's attacks were of the same series as those used by terrorists who hijacked an Egyptair plane in Athens last month, leading to the death of 59 persons when commandos stormed the plane in Malta. Officials said there is strong evidence that the Egyptair attack was an Abu Nidal operation.

The State Department said Abu Nidal's organization was held responsible for 50 terrorist incidents of the past eight years and about 12 in the past six months alone.

Many of the targets of assassination attempts or other attacks by Abu Nidal groups were Israelis, Jordanians or moderate Palestinians connected with the Arab-Israeli peace process. The most famous attack attributed to Abu Nidal, the June 1982 attempted assassination of Shlomo Argov, the Israeli ambassador in London, was used by Israel as the basis for its invasion of Lebanon.

Nine of the 12 most recent attacks attributed to the Abu Nidal group have been in Western Europe, especially in countries where terrorists from this group are being held in prison for earlier crimes. Both Italy and Austria have imprisoned Abu Nidal operatives, an official said.

Syria is culpable to some degree for support of Abu Nidal, but the group's base has shifted increasingly to Libya over the past six months, according to State Department officials.

The Reagan administration has long been at odds with the policies and international activity of Libya's leader Muammar Qaddafi. State Department officials said yesterday they knew of no specific warning to Qaddafi through diplomatic channels about Libya-based activities of Abu Nidal.

One U.S. offficial said it is "conceivable" that the attacks were launched to undermine the prospects of peace in the Middle East at a time of delicate negotiatons.

Pentagon officials also continued to monitor events today. The aircraft carrier USS Coral Sea, part of the Sixth Fleet, remained anchored in Naples rather than steaming toward trouble spots in the Mediterranean as it has in other crises, officials said.

There were reports that some State Department officials, frustrated by the latest round of terrorism, initially favored some kind of U.S. military response. But Pentagon officials, in informal discussions with their State Department counterparts on Friday, counseled against such action.

A senior administration official, explaining the rationale of those urging U.S. military reprisal, said it is less important to punish the terrorists responsible for the attack than the countries that harbor and train them, such as Syria, Libya and Iran.

"You have to assign responsibility somewhere," the official said. "The only way to stop this is to make it too costly for those countries to harbor terrorists. You have to go after the base."