Behind the facade of President Reagan's tough talk and unilateral economic sanctions against Libya, his administration remains divided and uncertain on a strategy to deal with state-supported terrorism.

Reagan came into office promising "swift and effective retribution" against foreign dictators who humiliate or harm Americans. But, with the sole exception of the capture of the Achille Lauro hijackers, he has proven unable to translate this promise into reality.

Reagan's difficulties reflect both strengths and weaknesses of his presidency. One "difficulty" for which we can be thankful is that he is a decent man who maintains civilized standards for retaliation. Reagan believes a response that endangers civilians is itself an "act of terrorism."

The president's political advisers learned that last summer when they pressed for retaliation after four U.S. Marines were killed in an attack on a cafe in El Salvador. When then-national security affairs adviser Robert C. McFarlane told Reagan that the options for military response probably would mean civilian deaths, the president refused to order a retaliatory attack.

Contrast this with what White House officials delicately call "an Israeli standard of proportionality," reflected in the Oct. 1 bombing of Tunisia where 60 Palestinians and 12 Tunisians were killed in retaliation for the murder of three Israeli civilians in Cyprus. Contrast it even more with the Jerusalem Post's account of how the Soviet secret police secured release of three kidnaped diplomats by the gangland-style murder of the relative of a radical Lebanese Shiite leader.

Unfortunately, the president also has faltered in combating terrorism for reasons of indecisiveness. The central problem is that the State and Defense departments have never agreed on a unified response to state-supported terrorism, and Reagan has never resolved the differences.

The roots of this conflict go back to the U.S. Marine involvement in Lebanon. Secretary of State George P. Shultz strongly favored this peace-keeping role for the Marines, believing it necessary to implement U.S. diplomatic policy in the Middle East. Defense Secretary Caspar W. Weinberger and the Joint Chiefs of Staff worried that the Marines would become targets.

This policy difference was resolved not by Reagan but by the horrible event of Oct. 23, 1983, when a suicide truck bomber hurtled into the virtually unprotected Marine headquarters in Beirut, killing 241 U.S. servicemen. The catastrophe sent political shock waves through the Republican leadership in Congress, where there were nightmares about Lebanon becoming Reagan's Vietnam. A few months later, Reagan withdrew the Marines from Lebanon.

But the cost of this involvement has haunted the administration's antiterrorist policy. Shultz, largely supported by McFarlane, became rhetorically bolder about responding to terrorism. Weinberger became even more skeptical about incidental use of force to support a diplomatic objective.

Reagan spoke loudly but carried a small stick. He became a specialist in such symbolic actions as the invasion of Grenada, ineffective sanctions against U.S. foes and the boast after the capture of the Achille Lauro hijackers that terrorists "can run, but they can't hide."

In fact, terrorists consistently do both in the Middle East. Reagan said in announcing the Libyan sanctions that "civilized nations cannot continue to tolerate, in the name of material gain and self-interest, the murder of innocents." But U.S. allies in Europe tolerate this all of the time, and the Pentagon opposes the kind of massive military action needed to defeat or discipline Libyan leader Muammar Qaddafi.

Reagan fills the policy void with public relations, at which he usually excels. But his elevation of economic gestures to the realm of high policy and his theatrical denunciations of Qaddafi exaggerate the importance of the Libyan strongman without undermining him.

The public acting out of U.S. frustrations with Libya could increase the pressure for military action and may even turn out to be good politics for the president. Swift and effective retribution it is not.

Reaganism of the Week: Describing Qaddafi at his news conference last week, Reagan said: "I find he's not only a barbarian, but he's flaky."