At the southern end of the Persian Gulf, the United States, France and Britain are sailing close to the edge of the Iranian-Iraqi war.

All three keep warships cruising constantly near the Strait of Hormuz, through which about two-thirds of the West's oil passes. And all three are involved in a complex game of parries, feints and taunts with Iran that edge close to confrontation.

The most pointed encounter recorded so far was on Oct. 18, when an Iranian Navy frigate tried to stop a French cargo ship sailing through the Strait of Hormuz into the Persian Gulf.

During the previous month, the Iranians had challenged, by their own account, more than 300 vessels, demanding to know if they bore cargo bound for Iraq. Often they stopped them and searched them; sometimes they seized them, and there was little the freighters could do. But this time the response was different.

The French container ship Ville d'Anvers did not stop for the Iranian Navy, according to diplomats in the Arab states on the gulf. Instead, a third ship came on the radio: a French Navy frigate.

"They just said, 'We will not allow you to intercept this ship,' " according to one military observer.

"As far as we know it is the first time the Iranians have attempted to stop a merchantman carrying the flag of a nation with a naval presence in the area," this source said.

The Ville d'Anvers' cargo remains unclear, but France has been a major arms supplier to Iraq.

"In typical Gallic fashion," said another observer familiar with the incident, "they thumbed their noses" at the Iranians.

The French warship and cargo ship sailed on untouched to the United Arab Emirates.

Paris had sent a direct message. The humiliation of the Iranian naval vessel, identified by sources here as a missile-bearing frigate, was carried out on the open international shipping frequency.

"Every man and his dog heard the dialogue that went on," one diplomatic source here said.

The Associated Press quoted France's ambassador to the United Arab Emirates as insisting that "France is determined that this [Iranian] business has to be stopped" and would "do all in its power to protect its interests in the region."

As Iran continues to stop other shipping, contacts among the contending sides here are usually made with more care to avoid any dangerous confrontations, but the contacts are frequent and tense nonetheless.

U.S. P3 airplanes hunting Soviet submarines and E2Cs monitoring the area refuel regularly at Omani military airports in which Washington has invested more than $300 million in recent years. On Oman's Goat Island, radar reportedly keeps track of everything that moves in the 30-mile-wide strait.

Observers who have traveled with shipping in the region talk of seeing Iranian surveillance aircraft flying over British warships, themselves tailed by U.S. jets.

According to western military sources, about once a week airplanes from Oman's small Air Force -- some of them believed to be piloted by British officers -- scramble to intercept Iranian and sometimes Soviet aircraft probing Oman's defenses.

Oman, apparently concerned about the extent to which it is perceived as a springboard for the western powers here, recently asked the Americans, British and French to cut back on visits by their warships to Oman's ports, according to a senior western diplomat. U.S. planes continue to fly in and out under a 1980 access agreement but are not allowed to base here.

The British presence consists of a destroyer and a frigate plus auxiliary ships. The French, operating out of Djibouti, on the Horn of Africa, have about the same. The United States keeps a carrier task force of five or six ships within a few days' sail of the area at all times, according to informed diplomatic sources here.

The Soviets, by contrast, have extensive naval facilities next door to Oman, on Socotra -- an Indian Ocean island belonging to Soviet-allied South Yemen -- and Aden, its capital.

"It is still thought the Soviets would stand to gain from cutting the West off to the gulf," one western diplomat said. But while the game of cat and mouse with Soviet submarines is a serious one, Iran's actions remain the center of concern at the moment.

The Iranian searches and seizures of civilian ships began in September after Iraq launched a series of massive attacks on Iran's Kharg Island oil facilities in August. Iran had warned that it might respond to such a move by trying to shut down the Strait of Hormuz completely.

Each side hopes to cut off the economic roots of the other in the five-year-old war fueled by the oil wealth of both nations.

At the northern end of the gulf, Iraq is waging a violent campaign against ships doing business with Iran. According to the Shipping Information Service of Lloyd's of London, at least 156 vessels have been attacked since May 1981, most of them with bombs or rockets, and most by Iraq.

But perhaps because Iran is now up against the threat of direct involvement by western powers in this region, it "has been pretty correct so far" about observing the rules of war when it stops ships, said one senior diplomat here.

Yoshi Tsujimoto, counselor at the Japanese Embassy here, confirmed this. He said that two Japanese-crewed vessels have been seized and taken to the Iranian port of Bandar Abbas for thorough searches during the past three months. "All the crews were treated very well," he said, but some containers bound for Iraq were confiscated.