The border between Morocco and Algeria swung open last week for the first time in seven years, raising cautious hopes for an eventual peaceful settlement of the Western Sahara conflict and leaving Libya increasingly isolated, in the view of Arab and western diplomats.

Since 1975, Algeria and Libya have jointly supported the Polisario movement fighting Morocco for control of the former Spanish Sahara. This prompted Morocco to break diplomatic relations with both countries.

But at the end of February, Algerian President Chadli Bendjedid and Morocco's King Hassan II met to discuss renewed diplomatic relations and possible peace talks on the Saharan conflict.

The Saharan issue has paralyzed the Organization of African Unity for more than a year, since the Polisario-proclaimed Saharan Republic became a member state. Two attempts to convene an OAU summit meeting failed last year, largely because of a boycott by Morocco and its allies.

In 1981, the OAU called for a cease-fire and a referendum among the Saharans to determine the territory's future. But the two sides have never agreed on how to conduct the vote, or even how many potential Saharan voters there are.

Western diplomats say the Moroccan-Algerian detente is a step toward stability in an important strategic region.

"Libya is the big loser," said one western analyst, adding that Libyan leader Muammar Qaddafi "may now be more isolated than ever before."

Morocco, Algeria and Tunisia have agreed in principle to closer economic and political cooperation under the label of a "grand Maghreb," the Arabic term for northwestern Africa. A senior North African official said the three-way axis would aim to prevent local conflicts such as the Sahara war that might draw in the superpowers.

He added, "Our unity might convince Col. Qaddafi to accept our principles of noninterference and mutual respect, and join us."

The economic burden of the war has increased for both sides because of falling oil revenues for Algeria and Saudi Arabia, which finance opposing sides of the war effort.

"The Algerians seem to be backing off financial support for the Polisario," said one western diplomat, "and the Saudis have pressed both sides to negotiate, telling King Hassan that there's no more money for the war."

Diplomats agree that another reason for the Moroccan-Algerian rapprochement is a prolonged stalemate in the war, following Morocco's completion last year of a 400-mile-long antitank wall around its coastal positions. Using a high-technology surveillance system supplied by the United States, the wall protects the so-called "useful Sahara," including towns and phosphate mines. The Polisario guerrillas have been relegated to the interior.

Neither side has indicated what compromise it might offer to break the diplomatic deadlock over the OAU peace plan. However, observers regard the progressive restoration of Moroccan-Algerian transportation and economic ties as evidence of a strong commitment by the two leaders to negotiate.

Western and Arab diplomats say the detente is part of an increasingly pragmatic Algerian foreign policy being pursued by Bendjedid, especially since Algerian-Libyan relations have declined. Last month, Bendjedid signed a friendship treaty with Tunisia.

Analysts in both Rabat and Algiers are uncertain, however, about Bendjedid's ability to win acceptance for an eventual compromise from both his domestic political base and the Polisario. They say Bendjedid will not abandon the principle of self-determination for the Saharans, as the Polisario might draw Libyan help in resisting such a move.

Observers in both capitals agree with an Arab diplomat who says Bendjedid can count on support from Algerian public opinion: "The Algerian people have never felt committed to the war as a truly Algerian struggle, and they don't think of Moroccans as their enemy."

But diplomats caution that Bendjedid's ability to impose a Sahara compromise on the National Liberation Front (FLN)--Algeria's only political party--will depend on the nature of the compromise.

"The FLN and the Algerian Army are both very strong lobbies, and both have hard-line elements," said one analyst. "Bendjedid must face an FLN congress at the end of this year and general elections at the beginning of next year. So it's possible that, from Algeria's side, a real solution may have to wait until after the elections."

The officials are more certain of Morocco's approach to a settlement. They point out that Hassan has staked his personal prestige on his claim to rule the Western Sahara.

"He could not accept any other outcome," said one diplomat. "But Hassan needs Bendjedid's assurance that Algeria will go along with a pro-Moroccan vote in a Western Sahara referendum. If Hassan got what he wanted from the Algerians, it's almost certain that he could impose his will on Morocco to make whatever concessions necessary."

The United States has refused to recognize Moroccan sovereignty over the Western Sahara, although it recognizes Morocco's de facto administration of the territory. While supporting the OAU cease-fire plan, the Reagan administration has sold increased numbers of weapons to Morocco.

That policy has drawn criticism from Democratic congressmen, and a House subcommittee this week sharply cut the administration's requested military aid for Morocco while increasing the developmental aid.

Some observers, however, view the Moroccan-Algerian talks as at least a partial vindication of the U.S. policy, which officials have described as necessary to give Morocco a sense of security that will permit it to negotiate.