A week after President Reagan decided to exert economic pressure rather than military might against Libyan leader Muammar Qaddafi, the Reagan administration has settled down to a long-term war of nerves with Libya.

Decisions made by Reagan last Monday and announced last Tuesday night exhaust virtually all U.S. economic means of pressuring Qaddafi to cease support of international terrorism. The most dramatic measure, a ban on nearly all U.S. trade with Libya, had been considered by Reagan five years ago but rejected as too drastic.

Because he has now exhausted most nonmilitary options for opposing Qaddafi, Reagan may face even more difficult decisions about use of force if, as widely anticipated by U.S. officials, terrorist incidents tied to Libya continue to take place.

"If these steps do not end Qaddafi's terrorism, I promise you that further steps will be taken," Reagan declared at his news conference last Tuesday.

Neither Reagan nor Secretary of State George P. Shultz would elaborate on "further steps," but Shultz pointedly told reporters two days later that "force is not always the best means, but it may be necessary on occasion."

In a move that reinforces U.S. military capability near Libya, the aircraft carrier USS Saratoga has been sent to the Mediterranean Sea, officials said yesterday. With the USS Coral Sea on station, this move will provide two aircraft carrier battle groups, with a total of 28 ships, in the Mediterranean for the first time in several months.

The increased force was described as providing more options to the United States if military action is ordered.

The departure from Libya of most of the 1,000 to 1,500 Americans there would also provide Reagan with more leeway to act militarily. Speaking to European reporters late last week, Reagan called these Americans "potential hostages" and said their departure under redoubled U.S. pressure is intended to "untie our hands with regard to whatever action might be necessary in the future."

Despite all of this, a military response to Qaddafi is by no means certain, whatever the provocation, because of grave problems that have bedeviled policy-making about Libya throughout the Reagan administration. Among these are:

*The difficulty, as in the attacks Dec. 27 at the Rome and Vienna airports, of pinpointing Libya's role in terrorist activities and of presenting a convincing public case. Syria and Iran have also been involved in training and logistics of shadowy terrorist groups.

*Persistent divisions in the administration about use of force. Defense Secretary Caspar W. Weinberger and career military personnel are known to be reluctant to accept risks and costs of military action against Libya.

The North African desert country of 3 million people would seem little match for the United States, even with its large stock of Soviet weaponry. Still, Libya is 4,000 miles away, and its newly acquired Soviet missiles could inflict painful losses in a U.S. air strike. A full military operation, according to a reported Pentagon estimate, could require six divisions, about 90,000 men.

*Lack of agreement on a clear national objective that would be advanced by use of power. Would use of force be a limited retaliation targeted tightly on terrorists' support facilities, as Reagan's pronouncements to date suggest? Or would a wider application of force be used to "cut out the cancer" of Qaddafi permanently, in the words five years ago of then-secretary of state Alexander M. Haig Jr.?

*Potentially far-reaching political repercussions on U.S. relations worldwide.

Western Europeans quickly made clear that they have little stomach even for economic sanctions against Libya. The Arab world, at least rhetorically, lined up almost unanimously on Qaddafi's side, in part after his energetic telephone lobbying. Concerned Middle Eastern diplomats warned the administration that U.S. policy is turning Qaddafi into a Pan-Arab hero.

The Soviet Union reacted sharply. Despite Moscow's apparent lack of enthusiasm about Qaddafi's unpredictability, a U.S.-Libyan clash presents a danger that the Soviet Union would provide even more sophisticated weaponry and, sooner or later, an increasingly direct military commitment that could bring about a superpower confrontation.

Libya was among the first problem areas discovered by the Reagan administration, and policy-makers have been agonizing about it ever since. On Jan. 21, 1981, the day after his inauguration, Reagan presided over a National Security Council meeting at which Libya was a main topic, according to Haig's memoirs.

State Department policy papers about "what to do" about Qaddafi and especially his support of terrorism were being written within a few days. By June 1981, an interagency study, persistently prodded by Haig, was under way. Within a few weeks, U.S. oil firms were being asked to evacuate their American employes and a covert CIA operation against Qaddafi was being discussed with congressional oversight committees.

In August 1981, U.S. and Libyan warplanes clashed over the Gulf of Sidra, which the Libyans insist is within its territorial waters, and two Libyan jets were shot down. The administration was well aware of the possibility of a military encounter when it authorized naval maneuvers in the disputed area.

An intelligence report in October 1981 that Libya had dispatched assassination squads against U.S. ambassadors in Western Europe galvanized further U.S. policy-making. One alleged target, Ambassador to Italy Maxwell M. Rabb, was hastily flown home.

Plans for administration sanctions, including a full ban on trade with Libya under the 1977 International Emergency Economic Powers Act, were under consideration. This action, finally taken by Reagan last week, was strongly opposed in 1981 by the Treasury Department under then-secretary Donald T. Regan, U.S. business interests and the State Department's economic and Near Eastern bureaus.

While confidential discussions of anti-Libya measures were under way, a less reliable intelligence report was received about Libyan "hit squads" being sent to assassinate Reagan and other officials.

Officials now say they believe that the "hit squad" report was erroneous.

The administration's initial sanction was a ban on use of U.S. passports for travel to Libya Dec. 10, 1981, and public appeals to Americans there to leave. Phase two, held in abeyance while Americans departed, banned importation of Libyan crude oil and was announced March 10, 1982.

Those sanctions had little effect on Libya, which switched its $5 billion in U.S. crude oil sales to European customers, with assistance from U.S. oil firms that continued working there. The administration decided that no dramatic or risky action would be taken until after the general election in November 1984.

On Dec. 13, 1984, the administration issued a new public appeal for Americans to leave Libya. This emerged from a new secret policy review and was intended to eliminate potential hostages in case of further U.S. action against Libya. The appeal was not widely heeded.

Another round of anti-Qaddafi policy-making that began last summer is reported to have resulted in a presidential authorization last fall for a CIA covert operation to undermine the Qaddafi regime. Military planning "of a precautionary nature" was also reportedly undertaken.

The most recent policy-making, touched off by the airport attacks, was the most ragged publicly, with misleading, contradictory and even erroneous statements issued by government spokesmen.

It took 11 days to promulgate the new measures. With those in place, Shultz said Thursday, the administration is "pretty much at the end of the road" on economic sanctions against Libya.