President Reagan last night virtually ruled out military retaliation against Lebanon for the hijacking of TWA Flight 847 and settled instead for a vow to launch a global attack on the roots of terrorism.

"Vengeance is not a satisfactory basis for policy," said a White House official in confirming that there were no plans to bomb or shell targets in Lebanon to avenge the hijacking and the murder of one of its passengers, Navy diver Robert Dean Stethem, 23, of Waldorf, Md.

Unmentioned but not denied by White House officials was the no-retaliation promise that Shiite leader Nabih Berri said Reagan had given to Syrian President Hafez Assad to break the impasse over the release of the 39 hostages from the hijacked airliner.

Reagan's decision against bombing or shelling Lebanon to punish the country for its role in the hijacking comes after 4 1/2 years of tough talk and appears to mark a switch to a broader and less bellicose attack against terrorism, with military reprisals only one of many options.

Reagan continued to talk of retaliation last night. In his brief, televised speech from the Oval Office announcing the hostages' departure from Syria for West Germany, Reagan said that "the murderers of Robert Stethem," the Navy diver killed by the hijackers, "must be held accountable, and that "those responsible for terrorist acts throughout the world must be taken on by civilized nations."

But before Reagan spoke, a senior administration official had acknowledged the difficulty of striking back at radical Hezbollah Shiites in Lebanon. "Hezbollah lives in urban areas," he said. "It is manifestly infeasible, and they know it, to conduct violent raids against them."

Military leaders on the Mediterranean and in the Pentagon since the hijacking of TWA Flight 847 had been reviewing possible targets in Lebanon in case the president ordered a retaliatory strike. But targets which were linked to the hijackers, isolated from population centers and big enough to make an impression on terrorists around the world proved hard to find.

"If we do anything," said one general familiar with the strike planning, "it should be something big. But if we did do something big, what American would dare travel by airplane anyplace in the world?"

Another concern, one expressed by a senior White House official, was that if the United States did bomb Lebanon the radicals there would grab a fresh set of Americans and hold them hostage. Many Americans are still in Lebanon, including businessmen, government officials, tourists and journalists.

Still other factors which stayed the president's hand in Lebanon even before he apparently gave the no-retaliation pledge to Syria were the difficulties in finding a target which would punish the radical Shiites in the Hezbollah, which the White House says were responsible for the hijacking.

As far back as January 1981, and as recently as last night, Reagan has been denouncing terrorism. He coupled many of his past denunciations with vows to punish the terrorists, saying in January, 1981, for example, that he would order "swift and effective retribution" if confronted with a hostage crisis like the one in Tehran which plagued former President Carter in 1980.

In October, 1983, after a lone terrorist had blown up the Marine compound at the Beirut International Airport, killing 241 servicemen, Reagan said: "Those who directed this atrocity must be dealt justice, and they will be." Last week, in reference to the hijacking of TWA Flight 847, he said: "Terrorists and those who support them must and will be held to account."

Asked about Reagan seeming to talk one way and act another when it came to avenging terrorist acts, a senior White House official said that to succeed, swift retribution "must be done within a climate that understands it and can distinguish between impulsive violence and purposeful sustained action over time." He added that "the base for sustaining firm action in the coming months" is emerging from the hostage crisis.

In 1983, as today, the administration ordered Navy warships to mass off Lebanon to provide Reagan the big stick of U.S. military power. After the terrorist attack on the Beirut airport, the Navy, with White House approval, planned a night bombing strike by Navy carrier planes. The bombs were loaded and the pilots briefed several times, but Reagan held them back.

One reason, officials said, was that Gen. P.X. Kelley, the Marine commandant, warned during Joint Chiefs of Staff deliberations on the retaliatory raid that his Marines in Beirut might be worse off after the retaliatory attack than before.

Last Tuesday, when negotiations to free the hostages seemed to be at an impasse, White House spokesman Larry Speakes took the unusual step of warning that Reagan was considering blockading Lebanon or closing the Beirut airport.

The United States at the time had the aircraft carrier USS Nimitz, the cruiser USS South Carolina and the destroyer USS Kidd within the battle group circling off the Lebanese coast that could have tried to establish a blockade.

But Reagan in the end opted for diplomacy -- and vowed after the hostages' release "to fight back" against terrorists in the future.