The United States and Libya, as the result of developments in the past month, are fighting an undeclared but increasingly violent war that administration officials say has three possible results.

Under the best outcome from the U.S. perspective, Libyan leader Muammar Qaddafi will be killed or ousted, with his successors in Tripoli abandoning the tactics of international terror.

A more guarded prospect envisions Qaddafi alive and in power, but his terrorism thwarted by western vigilance and economic pressures.

The third and most disturbing prospect is the reason much of the world now has a first-class case of the jitters: the spiral of violence increases, with Qaddafi-inspired terrorist attacks continuing on Americans and with President Reagan retaliating with increasingly forceful bombardments, blockades or even an invasion of Libya.

Thus far, the world's newest and perhaps most dangerous war has been conducted on one side by terrorists with 10-pound packages of plastic explosives and on the other by waves of U.S. warplanes, which last week bombarded Tripoli and Benghazi with more than 100 tons of bombs and missiles.

The disparity in military means is only one of many strange aspects of hostilities between the United States and a Third World desert country 5,000 miles away that is peripheral to any U.S. national interest except one: the security and safety of American citizens, diplomats and military personnel.

"A great power can't easily go to war with a small power," said retired general Bruce Palmer, former U.S. Army vice chief of staff and a historian of the Vietnam war. Last week's bombing of Libya, Palmer said, "is reminiscent of how we tried to punish Hanoi and make them cease and desist" guerrilla warfare and other attacks in South Vietnam through carefully calibrated U.S. bombing. "I thought we'd learned our lesson, but obviously we didn't," he said.

The war was triggered, so far as Libya was concerned, by a March 24 clash arising from U.S. naval exercises in nearby international waters, which Libya adamantly claims. In the face of a massive and calculated show of U.S. military power, Libya launched antiaircraft missiles and patrol boats. U.S. forces attacked the missile sites and sank the boats.

For the United States, the causus belli was a coded message from Tripoli March 25 authorizing Libyan diplomatic missions around the world to mount attacks on Americans. Qaddafi virtually announced his intentions that day in Tripoli when he proclaimed, "It is a time for confrontation -- for war. . . . If they [U.S. officials] want to expand the struggle, we will carry it all over the world."

The March 25 secret message was the first time, according to U.S. officials, that Qaddafi had issued a general order through Libyan diplomats to attack Americans. It drew an immediate response from the State Department, which began consulting other governments quietly and announced March 26 that Libyan agents were conducting surveillance against "American installations and interests around the world" in ways which suggested "that Americans are targeted for attacks."

When that March 25 Libyan message and follow-up instructions, which the administration says were intercepted, were translated into death and injury of U.S. service personnel in a West Berlin discotheque April 5, Reagan decided to retaliate with U.S. warplanes.

A crucial point in the April 14 U.S. bombing raid was the declaration by Reagan that "if necessary, we shall do it again." Thus last week's U.S. raid could well be not an isolated episode but the beginning of a long confrontation.

The U.S. military action last week has some elements in common with previous American experience, including U.S. raids on Libya at the time of the Barbary pirates in the early 1800s, U.S. retaliatory bombing of North Vietnam in the early phase of the Vietnam war and U.S. ground combat action based in part on the protection of American citizens in the Dominican Republic in 1965 and Grenada in 1983.

However, the current U.S. conflict is different in important respects from all earlier models, which makes them of little use to officials studying the three major near-term possibilities.

The early overthrow, assassination or death of Qaddafi and his lieutenants in U.S. military raids, leading to a reversal of his policies, would be a resounding defeat for state-sponsored terrorism. In the administration view this would have major impact, especially in the Middle East, and bring domestic and international political gains across a broad front similar to those arising from Grenada several years ago.

A clear if less dramatic success would be the frustration of Libyan-sponsored terrorism through heightened security measures, improved intelligence and increasingly tough economic and political sanctions by a united western alliance.

Administration officials said that if such a combination of measures can be achieved with currently reluctant allies, it might bring a change in operational policy, though not a change of heart by Qaddafi.

Under the third and darkest scenario, however, a continuation of dramatic Libyan-sponsored attacks on U.S. targets would lead to more extensive American bombing raids or other U.S. military actions, such as a naval blockade of Libya or use of U.S. or allied ground troops. Such a deepening struggle has the potential for enflaming radical states and splinter groups to new acts of terrorism. If Americans are harmed by such acts, the Reagan administration would find itself facing yet another series of questions about how to respond appropriately.

This "worst case" possibility for the Libya conflict could produce increasingly serious disputes within the Atlantic alliance, heightened political tension and potential military tension with the Soviet Union and increased instability and disaffection from the United States in moderate Arab states and other Third World nations. As military action and reaction continue, public support in the United States for administration policy toward Libya could erode.

Notably absent from the present conflict is diplomatic contact with Libya in search of a settlement of differences, despite or perhaps due to the fact that Secretary of State George P. Shultz has taken a dominant role in fashioning U.S. policy. The Reagan administration withdrew all U.S. diplomats from Libya, ejected Libyan diplomats from Washington and has rejected overtures for contact with Qaddafi. The U.S. ambassador to the Vatican, William A. Wilson, was reprimanded for holding unauthorized discussions with Libya in January.

In contrast to Reagan's words and deeds, President Jimmy Carter took a different approach when faced with intelligence reports in 1977 that Qaddafi had ordered the assassination of the U.S. ambassador to Egypt, Hermann F. Eilts. Carter notified the Libyan leader through diplomatic channels that the plot had been discovered and that Qaddafi would be held "personally responsible" if Eilts were harmed. Eilts, now a professor at Boston University, said last week that one "hit team" was arrested in Egypt and Qaddafi is believed to have recalled another group and to have dropped the assassination plan. At Carter's insistence, Eilts said, the episode was not made public and came to light years later.

In terms of military muscle and economic resources, there is no comparison between the United States and Libya, a backward if oil-rich nation of 3 million. As the United States learned in earlier conflicts with Third World nations, however, usable and effective power cannot be determined by gross national product or numbers of aircraft carriers.

Barry M. Blechman of Georgetown's Center for Strategic and International Studies and author of a book on the uses of U.S. military force since World War II, said a forceful response to state-supported terrorism is justified and necessary but that "the way the United States went about it was quite odd."

Blechman noted that two carrier task forces, which the Pentagon said involved 17 naval vessels, 155 warplanes and 14,700 U.S. military personnel, were required to drop bombs on a limited number of Libyan targets. To employ such forces costing billions of dollars to retaliate against terrorists wielding 10-pound plastic bombs shows "the muscle-bound nature of the U.S. military," as well as "the U.S. inability to employ limited covert operations" and other alternative means, he said.

The rules of international conduct among nations changed several years ago "when normal, civilized states began to be subject to planned nonstate and state-supported terrorism," said former undersecretary of state Lawrence S. Eagleburger. He approved last week's U.S. action because "we tried for the better part of five years to get the European allies to agree on a set of nonmilitary sanctions and they wouldn't."

Eagleburger said he is concerned, though, about divisions in the Atlantic alliance and especially about U.S. frustration and anger with European countries that refused to cooperate.

Prof. Robert E. Osgood of Johns Hopkins School of Advanced International Studies, who was a senior State Department policy planner in 1983-85, said, "Terrorism directed at Americans abroad has become the functional equivalent of the aggression of states."

Osgood expressed concern that due to the disparity between the Libyan and U.S. means of waging war, "the United States has defined a battlefield in which the other side has most of the advantages."

The circumstances that gave rise to the U.S.-Libyan dispute and the actions taken in recent weeks were foreshadowed in striking fashion by a 1977 Rand Corp. report for the U.S. Air Force, "Military Implications of a Possible World Order Crisis in the 1980s."

The report by Rand analyst Guy J. Pauker outlined the possibility in the 1980s of "a breakdown of global order as a result of sharpening confrontation between the Third World and the industrial democracies" that could bring about increasingly nationalistic responses, lawlessness, chaos and anarchy in world affairs.

The United States would be expected to use its forces to protect others and its interests and citizens as order breaks down, Pauker wrote. Such a U.S. role would raise such difficult choices as whether the United States should "be prepared to project its power into all parts of the world where Americans may wish to travel, trade, study and engage in any other normal and peaceful activity. . . . If not, where should one draw the line?"