Mauritania has officially ended its role in a desert war over the phosphate-rich Western Sahara by signing a peace accord with guerrillas struggling to set up an independent Arab state on the territory.

The decision by Mauritania, the westernmost Arab country, to come to terms with the Polisario guerrilla organization effectively dismantled an alliance with Morocco to battle the guerrillas and keep the territory under joint Mauritanian and Moroccan sovereignty.

The Moroccans have, however, indicated that they will continue the war and may annex Mauritania's sector of the Sahara to keep the Polisario from taking over.

At stake is a stretch of rock and sand whose estimted 80,000 inhabitants make it one of the world's most sparsely populated areas. Bordered by the Atlantic Ocean to the west, the territory is coveted for what lies beneath it: phosphates, iron ore and possibly vast shale-oil and uranium reserves.

The obscure desert war has drained the budgets of Mauritania and Morocco and exacerbated tensions in northwest Africa, particularly between King Hassan's government in Morocco and the radical Arab government of Algeria, which has been strongly supporting Polisario.

The Mauritanian decision affords the Polisario an opportunity to garner more international recognition by proclaiming a government for the southern third of the Western Sahara, to which Mauritania is renouncing all claims. At the same time, the move further erodes Morocco's claims to the rich northern two-thirds of the territory following a call by the Organization of African Unity last month for a U.N.-supervised resolution on the Sahara's future.

The agreement between Mauritania and the Polisario was signed Sunday night after three days of negotiations, according to news reports from Algeria. The pact was published in Algiers yesterday.

The agreement said Mauritania had decided "to abandon definitively the unjust war in the Western Sahara in accordance with measures determined jointly with the representatives of the Sahara people, the Polisario Front." These measures were not disclosed.

Nor did the text say when Mauritania would withdraw from its zone in the former Spanish colony, which Spain ceded to Morocco and Mauritania in 1975. No mention was made, either, of the estimated 10,000 Moroccan troops still posted in the Mauritanian zone of the Sahara and in northern Mauritania itself.

Morocco has warned Mauritania against pulling out of the Sahara and said it would take steps to ensure its own security in such a case. The Moroccan press has elaborated by saying its government then would have the right to annex the Mauritanian sector.

Morocco, which has borne the burden of the conflict since Mauritania agreed to a cease-fire with Polisario a year ago, appears determined to retain the territory despite the costly guerrilla war that is believed to be draining $1 million a day from the Moroccan economy.

In addition to its phosphate riches, the Western Sahara is believed by some experts to contain the world's largest reserves of uranium. Morocco, which already ranks as the world's largest exporter of phosphates, is eager to become a producer of the potentially more valuable energy source.

The Carter administration has remained neutral on the conflict over the territory, but has been urged by some congressmen to support King Hassan more strongly and sell him weapons that he wants to battle the guerrillas.

A State Department official said it was unclear how Morocco would react to the Mauritanian decision to pull out of the war, but that it seemd unlikely to herald an overall settlement.

"It's hard to see how Morocco and the Polisario can come to an agreement," the official said. "Their positions are pretty irreconcilable." CAPTION: Map, no caption, By Dave Cook - The Washington Post