The province of Dhofar, the southernmost region of the suanate of Oman, is a remote and primitive land where camels eat sardines, frankincense is a cash crop and dolphins frolic along the empty white sand beaches.

For more than a decade it was also the battleground for a guerrilla war in which Marxist-supported Dhofari rebels fought the forces of the sultan and his allies, including Britain and Iran.

That war is over now. The rebels lost. The government, which a few years ago controlled little more than this dusty provincial capital, is bringing roads and public services into hill regions and coastal villages that have never known such things in the past. A few dozen hard-core guerillas still roam the hills north of here, but there has been no real fighting for many months.

Dhofar remains, however, a potential flashpoint for the political disputes and military confrontations that threaten the stability of the Arabian Peninsula and the Persian Gulf, the source of most of the oil for the non-Communist world.

Some British officers assigned to the sultan's armed forces predict that a new uprising against the sultan will occur, if not guerrilla warfare in Dhofar then urban terrorism in the booming coastal towns around Muscat, Oman's capital. Separatist sentiment among the estimated 150,000 Dhofaris remains strong, Dhofari and foreign sources say.

The central government admits that it has not yet consolidated its authority over the hinterlands of the Indiana-sized province. The Marxist-dominated people's Democratic Republic of Yemen, the official name of the South Yemen, backed the rebels in the war. It has East German and Cuban advisers with its army and remains a hostile neighbor.

An Iranian force of some 4,000 men that fought with the sultan's forces against the rebels remains in Dhofar, equipped with American-made tanks and F-4 Phantom jets. Some British officers with the Omani army are questing whether there is any further need for the Iranians, who lost a Phantom to South Yemeni missile fire near the border in November, but Sultan Qaboos Bin said, Oman's ruler, says he wants them to stay on.

"As long as we aren't really sure of the intentions of South Yemen, of the Cubans and the East Germans there, we are not quite prepared to ask our friends to leave," the sultan said in an interview.

Oman and Iran face each other across the Strait of Hormuz, at the mouth of the Persian Gulf, the vital oil waterway of which Iran considers itself the guardian.

"The security of this country concerns Iran as much as us," the sultan said. "There is no doubt that as long as I feel I need their backing, I will have it."

The British, whose withdrawal from most of the Persian Gulf in 1971 stirred Iranian fears of upheaval and instability on the Arabian side, are schduled to pull out in March from two air bases that they have continued to maintain in Oman.

Planes from these bases flew occasional air support missions during the Dhofar war, but the sultan said he did not believe that the British departure would weaken his security. British officers will remain on duty with the sultan's military forces.

Oman and South Yemen are thinly populated countries with small armed forces.

Besides the Iranians, the sultan can summon up a total military force of some 15,000 men, many of them Baluchi warriors recruited from Pakistan. The Dhofar bridgade, which keeps watch over the lonely mountain outposts and desolate border checkpoints of the south, number 8,500, according to officials.

The guerrillas, known as the Popular Front for the Liberation of Oman, probably never had more than 1,500 fighting men, according to military intelligence estimates. Yet it took more than a decade, and at one time consumed nearly half the annual Omani national budget, to suppress the rebellion.

Said Saadoun, chief of Rakhyut, village between Salalah and the South Yemen border that the guerrillas conrolled for some time, had a simple explanation for the urnabout in the military situation.

"At one time the Communists were very strong," said. "But the sultan's armed forces and the friendly Iranians were stronger."

It was, of course, much more momplex. The military campaign the guerrillas succeeded, according to informed analysts, for reasons that went beyond military force and somewhat resembled the pacifcation program in Vietnam.

The rebellion began as an uprising against the former sultan, Said Bin Taimur, a tyrannical and reactionary despot who kept Dhofar and indeed all Oman in conditions of appalling backwardness.

His son, the present sultan, ousted him in 1970 and embarked on a modernization and liberalization program that gradually defused some of the opposition. Defectors from the rebel ranks - nlown as Firquas, began working for the sultan. In the words of an official briefing paper, "their unparallieled local knowledge and anative cunning have been invaluale in outwitting the enemy."

Abdel Khalek Mansour, regional administrator of the Ministry of Communications, a Dhofari who fled the old sultan's rule, said the army by itself could never have defeated the insurgency.

Although the rebels had been trained and equipped by Communist cadred from outside, he said, the guerrillas themselves were motivated mostly by antipathy to the old sultan and when he was removed they began to return to the government side.

The Firqas, the regular armed forces and the Iranians mounted coordinated campaigns that induced hundred of the guerrillas to surrender and drove most of the others into South Yemen. The sultan proclaimed victory in December 1975.

Spaoradic fighting continued, how-Arabia, which for years had sought to ever, until last spring. Then Saudi isolate South Yemen and prevent the spread of Marzism in the Arabian Peninsula, suddenly entered into an agreement with South Yemen government.

The Saudis recognized the South Yemen regime and are understood to have offered economic aid to the impoverished country. In exchange they apparently insisted that South Yemen stop supporting the rebels in Oman.

"Saudi Arabia is trying its best to make sure the Arabian Peninsula is free from trouble. We welcome and bless this," the sultan said.

While the border remains tense, the sultan said, "South Yemen is not making any troubled for us at the moment."

In five days of traveling around Dhofar, I saw no evidence of any military action. My request for permission to visit South Yemen and talk to officials of the Dhofar rebellion who still work there was turned down by South Yemen.

From the point of view of the conservative governments and their Western allies, the suppression of the Dhofar rebellion is a major step in a series of events that has considerably improved security in te gulf in the past three years.

Iran and Iraq settled their dispute over the Kurdish rebellion; the United Arab Emirates survived a potentially disruptive dispute over their continued federation; and Saudi Arabia used its money to influence South Yemen.

That leaves the Iraq-Kuwait border dispute and the still-tenuous situation in Oman as potential sources of trouble.

A paper submitted to Oman's development council by the Dhofar governor, "his excellency the Wali," last month summarized the economic development projects undertaken since the fighting ended. Given the primitiveness of Dhofar, it is a list not of smelters and petrochemical plants but of water holes dug and roads built.

"Once achieved," these projects "should contribute to the development of a larger objective - the introduction of the effective authority of his majesty's government throughout Dhofar," the paper said.

Some doubt remains, however, about whether that can be permanently achieved.