The Reagan administration said today it would be "fine with us" if Israel takes military action against the sources of last Friday's terrorist attacks in Europe, and accused Libya of being the prime supporter of the renegade Palestinian group that is being held responsible for the airport violence.

Administration officials in Washington and California, where President Reagan is vacationing, appealed to governments in Europe and elsewhere to join the United States in exerting diplomatic and economic pressure against Libya. The officials also pointedly noted that a U.S. "military option" against the sources of terrorism has not been ruled out.

In New York tonight, the U.N. Security Council unanimously condemned Friday's attacks but called for restraint in responding to them.

Today's administration statements drew a distinction between military responses to international terrorism that might contribute to "an escalating cycle of violence" in the Middle East and military responses to terrorist acts "in an appropriate, measured and focused way." The administration opposes the former and endorses the latter.

"If we can find out who they [the terrorists] are, or if another nation can find who they are, and they attack them and wipe them out, that's fine with us," White House spokesman Larry Speakes said in Palm Springs.

Speakes' statement represented a change in tone from White House remarks on Saturday calling for Israeli restraint following attacks on Israeli airline counters in Rome and Vienna. Eighteen persons, including five Americans, were killed. However, Speakes denied any policy change.

A senior White House official said the basic message is unchanged: "We're saying to Israel that we understand you have to take action against terrorism and we approve of that if it is directed at those responsible for the act, rather than civilians.

Officials said the purpose of the U.S. pressure on Israel was twofold. The United States wants to prevent bombings of Palestine Liberation Organization camps in Jordan or elsewhere, because there is no evidence linking the main body of the PLO to the terrorist acts and because such strikes would inevitably damage the Middle East peace process, officials said. They added that there were U.S. concerns that the Israelis would use the attacts as a pretextfor wiping out surface-to-air missiles that Syria has moved into Lebanon.

But administration statements Saturday were misinterpreted in some quarters as being "soft" on terrorism, officials said. They said that today's statements were designed to balance concern about the impact of portential Israeli reprisals with presidential approval of "appropriate" retaliation.

"We have always been firmly opposed to an escalating cycle of violence, which contains the seed of broader and more devasting hostilities," the White House and State Department said in coordinated statements today. "In that contentions which can only feed that cycle. At the same time, it has been our firm policy that terrorism cannot go unanswered. We have always retained the right to respond to terrorist acts in an appropriate, measured and focused way."

While evidence is still being analyzed, the administrtion statements said, "all of the indications we have so far" point to the terrorist group headed bu Abu Nidal as the perpetrators of last Friday's attacks. "Other governments directly concerned share this judgement," the statements added.

The Abu Nidal group of some "several hundred" people has been described by administration officials as among the most lethal, experienced and best organized of terrorists organizations. Abu Nidal was expelled from the PLO in 1974 after threatening to kill Chairman Yasser Arafat.

State Department spokesman Charles E. Redman cited "the indiscriminate attack, the choice of targets, the preliminary evidence and the method of operation" in the recent terrorist assaults as all pointing to Abu Nidal. "These are murderers who go out of their way to target civilians and have attacked and killed many Arabs as well as Israelis, Americans and Europeans," Redman said.

Abu Nidal has recently been in Libya and "much of his operational base" is now in that country, Redman said. He added that "there is also apparently an Abu Nidal presence in Syria."

The State Department's report on "Patterns of Global Terrorism" for 1984 said the Abu Nidal group was then based in Syria and Syrian-controlled Lebanese territory. The 1984 report described Abu Nidal as "a semi-independent actor in the Middle East morass" although it said that his attacks had been mounted "usually with Syrian foreknowledge, if not complicity."

Redman said yesterday that Libyan leader Muammar Qaddafi has given Abu Nidal "a considerable amount of financing and assistance." Other officials noted as a direct and public link a report from Tunis, attributed to a senior official of the Tunisian Ministry of the Interior, that two Tunisian passports used by gunmen who attacked the Vienna airport had been confiscated by Libyan authorities in August and September from Tunisian migrant workers.

Redman noted that Libya publicly hailed last week's airport attacks and that Abu Nidal was quoted in an interview last October from Libya as saying that Qaddafi "is a great help to us."

Noting that the administration has taken a number of steps since 1981 to exert diplomatic and economic pressures on Libya, the White House and State Department spokesmen called on other governments to join in such actions. In addition to these pressures, Reagan is reported to have authorized covert Central Intelligence Agency actions aimed at undermining the Libyan regime.

State Department officials expressed hope that Italy, which has major stakes and influence in Libya but has been reluctant to oppose Qaddafi, will change its stand now that the Rome airport has become a target of Libyan-supported terrorism.

The U.N. Security Council statement passed tonight condemned the attacks as "unjustifiable and criminal" but cautioned governments to "exercise restraint" in responding to them.

The United States, which proposed the council statement this morning, acceded to demands from council members belonging to the nonaligned movement for the language cautioning against military retaliation.

American representative Herbert Okun was forced to accept the nonaligned members' amendment, which was a response to the White House and State Department statements reserving Washington's right to use military force against terrorists.

The United States had asked council members to urge that those responsible for "these deliberate and indiscriminate killings be brought promptly to justice." But in the end, the council urged that they "be brought to trial in accord with due process of law."

The nonaligned nations also added to the statement a call "upon all concerned to exercise restraint and refrain from any action inconsistent with their obligations under the U.N. Charter and other rules of international law."

American officials said they went along with the changes on the understanding that these limits apply to all nations and not solely to the United States and Israel.

The council adopted a similar unanimous statement on Oct. 9 condemning the hijacking of the Italian cruise ship Achille Lauro and the murder of an American passenger.