JEDDAH, SAUDI ARABIA, AUG. 28 -- Incensed by last month's violence at Mecca, Saudi Arabia has resolved to confront the Islamic revolutionary government in Iran and is actively seeking to isolate it in the Middle East and the Moslem world, a well-placed Saudi source said today.

The new Saudi determination means at least a temporary shift from the kingdom's attempts to maintain relations with Ayatollah Ruhollah Khomeini's militant Islamic government and avoid provoking its hostility.

This change in policy, although motivated by clashes between police and Iranian pilgrims July 31 in Mecca, also coincides with and could affect the U.S.-Iranian confrontation in the Persian Gulf, which involves coordination of Saudi Arabian and U.S. military forces.

Saudi outrage after Mecca has given rise to an uncharacteristic campaign to marshal public opinion in Islamic countries and the West against Iran and a diplomatic effort to persuade the Arab League to unite behind Iraq in the seven-year-old gulf war.

The Saudi source said it could lead eventually to a formal break in diplomatic relations with non-Arab Iran, unilaterally or as part of an Arab League decision.

"I think our main objective now is to get the Iranians on the run, to put them on the defensive," said the informant, who has access to thinking at the highest levels of the Saudi monarchy.

The Saudi foreign minister, Prince Saud, won an Arab League decision this week demanding that Iran comply by Sept. 20 with a U.N. cease-fire appeal.

Although Saud had sought a tougher league stand, Saudi officials said the decision nevertheless marked the first time Iran's Arab friends, Libya and Syria, have joined fellow Arabs in fixing responsibility for ending the conflict on Iran.

The Saudi decision to take the diplomatic and propaganda initiative against Tehran has not yet expanded to affect oil policy, potentially the strongest Saudi weapon against Khomeini's government, the source said.

With ability to produce up to 12 million barrels a day, Saudi Arabia could flood the market and drive down prices, cutting into the oil revenues that Iran depends on to finance its costly war effort against Iraq.

Some sources have suggested Saudi Arabia already has begun selling more than its quota agreed on in the Organization of Petroleum Exporting Countries as part of the new drive against Iran. But the Saudi source denied this, saying that would be like "shooting ourselves in the foot" since Saudi Arabia also wants prices maintained.

Industry analysts said August output by OPEC countries could rise 3 million barrels a day above the organization's 16.6 million ceiling, which was designed to maintain an $18-a-barrel minimum price. The overproduction has caused a drop in prices in recent days, leading to scheduling of a special OPEC meeting Sept. 7 in Vienna.

The U.S. military buildup in the gulf, along with the Mecca confrontation, has led to increased U.S.-Saudi military cooperation, the Saudi source said. He declined to be specific.

According to reports from Washington, the Reagan administration has obtained expanded landing facilities for carrier-based U.S. warplanes in a loosely defined and secret agreement. In addition, advanced Saudi surveillance planes reportedly have worked more closely with U.S. forces in monitoring Iranian Navy and Revolutionary Guard vessels.

"Operationally, what needs to be done is being done, and if it wasn't, you could not be doing the things that are being done (in the gulf)," said the Saudi source, who is particularly well informed on U.S.-Saudi military cooperation.

"When you have an armada out there and we have 40 years of relations . . . , " he added. "I don't think it's difficult to imagine that there is cooperation."

Some U.S. congressmen have complained that Saudi military cooperation in the gulf buildup is inadequate, particularly the refusal to move into open political endorsement of U.S. policy in the region and participate publicly in it.

These considerations have become particularly important against the background of administration plans to try again to win congressional approval for a large arms sale to the kingdom, reportedly including additional F15 aircraft.

Congress has refused such requests in the past, accepting Israeli arguments that the modern U.S. arms could diminish Israel's military superiority over Arab nations. Putting the Saudi arms request in the context of confrontation with Iran, therefore, could bolster administration and Saudi positions in anticipated congressional debate over the sale.

"We cannot afford a slap in the face by the U.S. Congress during this crisis," the source said, referring to earlier congressional refusals. "Nor can you afford to be seen giving us a slap in the face during this crisis."

Four U.S. Airborne Warning and Control Systems planes (AWACS) have been operating since 1980 from Riyadh, along with three aerial refueling tanker planes, to provide military surveillance in the gulf area.

Some 600 U.S. personnel have been based there to service them and Saudi Arabia's own AWACS radar planes, operated jointly with the United States as part of a training program, the informant pointed out.

But Saudi Arabia has refused the political embrace that would be implied in stationing U.S. fighter aircraft on Saudi soil. U.S. air power for the gulf operations, as far as has become known, generally has been restricted to carrier-based planes flying from the Sea of Oman or land-based facilities granted by Oman.

The hardened attitude toward Iran here could ease Saudi reticence to be seen as part of the U.S. effort. But Saudi leaders have been careful to avoid any indications of this on the record.

King Fahd's government, like those of his predecessors, has consistently chosen accommodation over confrontation and avoided identifying itself too closely with Washington, Israel's main benefactor. It was unclear, therefore, how long the new Saudi activism against Iran would last or how far it would lead the kingdom in cooperation with U.S. military policy in the gulf.

The source said Saudi Arabia could be less reluctant to make such a gesture if the administration's gulf commitment turns out to be a long-term policy, in which U.S. military power is applied in a consistent way.

"People were not sure you were not going to do the Beirut Marine trick on them, pack up and go home again," he said.

The Reagan administration, after saying U.S. interests demanded shoring up the Lebanese government, quickly pulled U.S. Marines out of Lebanon when their barracks were blown up by truck bombs attributed to Lebanese Shiite groups with close ties to Iran.

The Saudi source, urging the United States to stand firm on its new gulf commitments, said Saudi Arabia has found that Iran backed down in several earlier potential confrontations in the gulf.

When Saudi fighters shot down an Iranian F5 in 1984, he said, Iran sent up more fighters in an apparent challenge to the Saudi Air Force. But when Saudi planes scrambled to meet the challenge, eventually putting more than 20 planes from each side in the air, the Iranian planes received orders to return to base, he said.