THE LATEST AMERICAN response to Libya's Muammar Qaddafi is a case study of the frustrations that great powers face when they come into conflict with small but dangerous Third World countries.

Since taking office in 1981, the Reagan administration has made containment of Qaddafi a top priority. During that time there have been sanctions and more sanctions, public discussion of covert-action programs, and the use of outright U.S. military force. Indeed, Libya was the first country against which the Reagan administration fired shots in anger when, on August 19, 1981, U.S. Navy fighters downed two Libyan jets that had challenged our right to fly over the Gulf of Sidra.

Yet an honest appraisal must concede that the U.S. policy toward Libya hasn't achieved its goals. Today, despite nearly five years of American attempts to contain Qaddafi, the Libyan leader seems as entrenched and dangerous as ever.

If nothing else, this depressing history offers some useful lessons for the future. Above all, it demonstrates the pitfalls of trying to conduct an assertive foreign policy in a democracy with diverse bureaucratic and foreign-policy interests. The bureaucratic and alliance arguments that emerged during the debates over Libya policy could never be separated from our overall Mideast policy wrangles and often seemed to overshadow the basic goal of dealing with Qaddafi.

During my four years as the senior Mideast specialist on the National Security Council staff, I attended dozens of meetings dealing with Libya and the Qaddafi problem. The issue came up often when I traveled with administration officials to Europe and the Middle East.

Although there was consensus within the administration, the Congress and amongst our European allies and Arab friends that Qaddafi was a menace and that his activities should be curtailed, the consensus evaporated when it came to recommendations about specific actions.

During private sessions, Arab and European statesmen were usually outspoken in their venom toward the Libyan leader. The Egyptians, in particular, had good reason to be deeply concerned about Qaddafi's subversive actions, both within their own country and against friendly neighbors such as Sudan and Tunisia.

But when the chips were down and decisions had to be taken, even the Arab leaders most threatened by Qaddafi found that they had to think carefully before seeking direct U.S. support, especially if a military dimension was involved. On several trips I made to Arab countries with Secretaries of State Alexander Haig and George Shultz and with National Security Adviser Robert C. McFarlane, we got the same contradictory message: total agreement with our assessment that Qaddafi was a threat, studied ambiguity about what we should do about it.

Similarly, European attitudes changed over the years from mild amusement about our "thing" over Qaddafi to cautious agreement that he was a problem and some remedy had to be found. Yet whenever it came down to the bottom line of "what shall we do about the man?" the silence was stunning.

Part of our problem was that each Libya crisis we faced was different. The Gulf of Sidra incident was unique -- and the most easily managed -- because it resulted from our initiative. We were in the driver's seat. But in other cases we had to react to Libyan actions that were completely unpredictable. We discussed many plans for dealing with Libya, but they rarely fit the precise set of circumstances that we actually faced.

The administration's internal bureaucratic difficulties on Libya policy, although not as antagonistic as the fight over Lebanon, were complicated. Reaching a consensus required that all the bureaucratic entities -- the White House, State Department, Defense, CIA and, on occasion, Treasury -- resolve their differences quickly. That hasn't been easy in this administration since the president's style of management has allowed the agencies and departments a great deal of autonomy.

Crises force the different agendas to the surface. The White House worries about the "big picture," including domestic policies, relations with Congress and, most important, what the president should say publicly -- since every word he utters on the subject will be flashed around the world in microseconds.

The State Department reflects the views of Secretary of State Shultz, who feels especially strongly about terrorism, but it also takes into account the views of the European and Near East bureaus, where any crisis over Libya raises a myriad of complicated and contentious questions concerning European and Arab relations.

At the Defense Department, both military and civilian personnel are influenced by practical questions -- what can we do and when can we do it -- and strategic considerations relating to our worldwide military posture, Soviet military activity and, in the case of the Middle East, our strategic ties with the moderate Arab states, which are critical to our defense of the Persian Gulf.

The biggest bureaucratic battles were those involving the use of military force to counter Mideast terrorism. It has often been said that the hawks in this administration are in the State Department and the White House and the doves are at the Pentagon and the CIA. Although this formulation grossly oversimplifies the issue, it is true that on certain Middle East issues (Libya, Arab-Israel, Lebanon) the Defense Department and the CIA have been the agencies most likely to highlight the political dangers of using American military force in the Arab world.

In my experience, the caution of the Pentagon and CIA stems from two different causes: concern that American military responses to Arab terrorism could jeopardize our military posture throughout the Mideast, and the long and bitter legacy of Vietnam and Watergate, which has had a permanent impact upon military and CIA professionals and caused them to be skeptical and cautious whenever administration officials suggested using military force.

In dealing with Libya, I came to learn that no matter how determined we might be, there were always reasons for caution. Indeed, when seasoned bureaucrats present the arguments for inaction, it isn't difficult for them to convince the president's closest advisers that any military retaliation against terrorists carries large political risks.

Consider, for instance, the most basic questions that always come up when military options are discussed. What is the purpose of retaliation? Is it to punish? To deter? To weaken or destroy the terrorists?

Each goal may require different methods -- diplomacy, economic pressure, overt force, covert force -- and each method has its own supporters and detractors. The regional bureaus in State invariably stress a preference for diplomatic action. You hear things like: "Let's bring our friends along," "Don't let's antagonize the Italians anymore," "Let's be sensitive to Mubarak's internal problems," "Let's remember our long-term goals in the Middle East."

As for economic sanctions, there is a long bureaucratic memory here and in Europe about such measures. Whenever sanctions are proposed, the skeptics argue that historically, such measures rarely work unless they are applied over time and with great vigilance by all major trading partners of the target.

Given this history of frustration and failure, what are the chances for the administration's latest proposal to use economic rather than military measures to pressure Qaddafi?

First, it wasn't any surprise that the president ruled out military force once again. The problems are evident. After the Vienna and Rome airport attacks, the preferred partners for military action should have been the Austrians, Italians and the U.S. -- an unlikely trio indeed.

From a military standpoint, a joint U.S.-Israeli attack has attractions. But one can be certain that a near-solid phalanx of State, DOD, CIA and NSC staffers would have developed a scary list of scenarios about how such a joint raid would weaken -- perhaps fatally -- our ties with Egypt and other Arab friends. Arab opinion would see Qaddafi as a victim and the action would be further proof of U.S.-Israeli collusion at the expense of the Arab world. While many Arab leaders might be happy to see Qaddafi toppled, anything short of his departure would cause them more pain than pleasure.

A military attack against Qaddafi runs some obvious dangers. An air raid against heavily defended targets in Libya risks the loss of American war planes and the capture of U.S. pilots. The administration remembers all too well the agonies this caused in late 1983, when Lt. Robert O. Goodman, Jr. was shot down over Lebanon and taken prisoner by the Syrians.

Attacking terrorist camps that are less well defended might reduce the risks of U.S. casualties, but it would increase the likelihood of killing civilians. Television pictures of such "crimes" would be made available with great alacrity to the world networks, all chomping at the bit for good footage for the evening news. If SAM sites or other military targets were attacked, East European or Soviet technicians might be killed. Or, perhaps more important, since there is bitter opposition to Qaddafi within his own military, there is a risk that such raids might kill those who have been plotting his overthrow in secret for months.

Some experts have suggested that Libya might retaliate against a U.S. strike by attacking NATO bases in Sicily that are within combat range of Libya's Soviet-made jets. There is also the possibility of attacks on U.S. ships in the Sixth Fleet. If the Libyans sank or seriously damaged an American warship, the U.S. would have to choose between escalation or humiliation -- neither of which is an attractive alternative. There is also the chance Qaddafi could hold hostage the 1,000 to 1,500 remaining Americans in Libya.

A more indirect and less violent method of applying force against Libya would be a selective naval blockade of all oil tankers departing from Libyan ports, thereby cutting off Qaddafi's dollar income. But to imagine such a quarantine being accepted by the West Europeans and obeyed by East Europeans is to hope for too much.

So, as in the past, the U.S. is left with the same tepid option of economic sanctions -- and the same question it has faced for five years. Short of going to war against Libya, does the U.S. have any realistic chance of solving the Qaddafi problem?

The answer depends on what the U.S seeks to accomplish. If the American goal is to topple the Qaddafi regime altogether, then sanctions won't do the job. But if the U.S. has the more modest goal of deterring Qaddafi from his most egregious actions, then sanctions might be successful.

Libya, after all, is highly vulnerable to economic pressure, especially with oil prices low and supplies abundant. Reducing Qaddafi's oil revenues wouldn't reform his character, but it would surely force him to cut back on development, arms purchases and funds for terrorists. The question, as always, is whether our European allies will cooperate. The indications this past week weren't encouraging.

The litany of the constraints operating in January 1986 is a depressing reminder of why it has been so difficult for the U.S. to grapple with the Qaddafi problem. The one clear requirement -- whatever the U.S. decides to do or not do in the future -- is for better intelligence. The Achille Lauro affair was a great success because the intelligence was so good, and because we had military forces in the right place at the right time. Without good intelligence, we won't be able either to preempt attacks or, failing that, retaliate for them.

We must plan to use more deadly methods to disrupt and destroy terrorist networks. For what we have seen so far from Qaddafi and his cohorts is only the tip of the iceberg.

There is a strong likelihood that we will one day have to deal with a "catastrophic" act of terrorism -- involving nuclear weapons, biological weapons or radioactive materials. If this happens in western Europe, governments there may at last be motivated to put aside their economic interests and join in concerted action. In the meantime, the priority in dealing with Qaddafi will be the irreducible need for intelligence and security.