Well-armed Moslem extremists holding hostages in the Great Mosque of Mecca withstood an assault by surrounding Saudi troops today and held firm in the second day of their occupation of Islam's holiest shrine.

The takeover of the sacred mosque, reportedly by Islamic militants who have not been further identified, constituted an unprecedented affront to this conservative Islamic kingdom whose rulers are responsible for Moslem holy places around the world and regard protection of the Islamic faith as one of their royal duties.

It brought home with brutal intensity to the Saudi royal family the potential for turmoil in the revolutionary brand of Shiite Moslem Fundamentalism that has surged to power in nearby Iran and whose influence is being felt in several wealthy Persian Gulf oil countries.

[Iran's charge d'affaires in Washington, Ali Agha, said yesterday that it is "totally untrue" that the Mecca incident is "in any way connected with Iran or Khomeini or the Shiites." He said that the incident "was the result of the action of one single man." He refused to elaborate, however, saying only that details "will come out."]

Saudi government officials showed considerable nervousness and embarrassment here in Riyadh, the royal capital. The Interior Ministry posted extra guards, armed with submachine guns and knives in their belts, around government buildings. The government restored communications with the rest of the world early this morning after a 15-hour blackout, but it issued only two sparse communiques on the takeover, blaming "a gang acting outside the Moslem religion."

Price Sultan, the Saudi defense minister, and his brother Prince Nayef, the interior minister, spent their second consecutive day at Mecca supervising operations around the occupied mosque, Saudi officials said.

Reports telephoned from residents of mecca, about 700 miles southwest of Riyadh, said gunfire crackled around the giant mosque during the day as Saudi troops unsuccessfully tried to work their inside without making a frontal attack.

Nayef emphasized in a communique tonight that Saudi troops surrounded the mosque and could easily recapture it. But "because of a desire to protect the innocent still inside the sacred house of God, and out of respect for the holiness of the sacred mosque profaned by these criminals," he said government forces were avoiding an all out attack for the time being.

[Reuter reported from Bahrain, quoting travelers from Mecca, that relatives of Saudi Oil Minister Zaki Yamani were among the hostages.]

Saudi officials declined to say how many troops were involved or which branch of the armed forces they came from. Reports of unusual movement by C130 transport planes near Jeddah, nevertheless, indicated reinforcements were being flown in. Sultans' presence in Mecca also led observers to believe elements of the 45,000 man Saudi Army were on the scene as well as Interior Ministry security police.

In addition, Prince Abdullah, commander of the 35,000-man National Guard that is the kingdom's chief internal security force, was reported to have returned during the evening from a private visit to Morocco to participate in decisions on the Mecca operations.

Prince Fahd, the crown prince who runs the kingdom's affairs under broad supervision from King Khalid, was in Tunis for an Arab summit conference. There were reports early in the day that he planned to return to direct the Mecca siege personally, but there was no confirmation that he had in fact come home.

The Saudi state television tonight showed lengthy footage of Fahd's meetings with other Arab leaders at the Tunis summit conference, with soothing music in the background. It made only a four-sentence mention of the Mecca takeover, calling the occupiers "renegades" and assuring the audience that authorities were protecting the holy site and the hostages inside.

There was no indication how many hostages were being held or what their nationalities were. The two Interior Ministry communiques, read repeatedly without comment over state radio throughout the day and evening, said only that they were "Moslems in the mosque for dawn prayers."

[Saudi sources in Washington said the situation around the mosque is "under control" but that it would take "some time" to clear it because of its great size. The sources emphasized that the takeover is "nothing political, only religious" and shied away from specifically identifying either those who staged the takeover or their hostages].

Reports from sources outside Saudi Arabia said a Moslem clergyman and two security guards were killed during the takeover early Tuesday. Saudi officials here declined to confirm these reports or provide casualty figures for the exchange of gunfire today.

Similarly, officials turned away queries about the number of the attackers, their identity, their goals or their possible connection to the U.S. Embassy takeover in Iran. Estimates of their number from foreign analysts ranged from 50 to 200. These analysts emphasized that their information was sketchy and based on second-hand reports.

Non-Moslems are forbidden to enter Mecca even in ordinary times and the city has been reported sealed off by security forces screening all travelers since the gunmen took control of the mosque at dawn Tuesday.

The reports that the attackers were Iranian emanated from Arab sources at the Tunis summit conference. Nayef said in his second communique only that there "are no indications that the incident is linked to any particular nationality." This raised the possibility that the attackers could be Saudi or other Arab Shiites -- or perhaps a mixed band -- since observers believed the Saudi government would have been quick to affirm they were not Saudis if this was the case.

There were some reports, beginning in Tunis, that those staging the takeover were Mahdists, a small sect with beliefs somewhat similar to Shiites. There are Mahdist sects within Saudi Arabia, especially near the border with Yemen.

Foreign sources have suggested, however, that the reports from Tunis about the Iranians seemed logical because of incidents elsewhere and the large number of Iranians who had been in Mecca for the Hajj, the Moslem pilgrimage.

Saudi authorities expelled a number of Iranian pilgrims during the annual Haji that ended last week because of their participation in demonstrations supporting the radical Islamic rule of Ayatollah Ruhollah Khomeini in Iran, reliable sources reported. But out of the 2 million pilgrims, it was easily possible for a number of Iranian radicals to have escaped that expulsion and stayed around to mount the mosque operation, they suggested.

There also was a strong indication of a Shiite Moslem connection in the first official communique, issued by the Saudi Interior Ministry shortly before 5 a.m. today. The gunmen, it said, brought forth one of their comrades and presented him to the hostages as the 'Mahdi," the "last prophet" that Shiite doctrine says will come as a messiah.

This belief is particularly Shiite. Most of Saudi Arabia's more than 5 million native inhabitants are members of the Sunni sect, the majority Moslem strain that regards Shiites as heretics.

The two main Islamic sects differ principally on the lineage from the prophet Mohammad, Islam's founder, and have few substantial doctrinal quarrels. Sunnis often look down on Shiites, however, and relations between Saudi Arabia and Iran were rarely warm even when Iran was ruled by the shah, an absolute monarch like the Saudi royal family.

Saudi Arabia is known to have played a role in efforts to secure the release of Americans held hostage at the U.S. Embassy in Iran. Fahd personally telephoned Yasser Arafat, head of the Palestine Liberation Organization, to seek PLO intervention with Khomeini and his followers, knowledgeable officials said.

The takeover also came on the first day of the Moslem new year, 1400, an observance regarded with particular importance by Shiite Moslems and generally treated casually by Sunni Moslems.

Saudi Arabia has its own small Shiite minority, estimated to number about 125,000. They are concentrated in the Eastern Province, the area of most of Saudi Arabia's vast oil resources that have made the kingdom one of the wealthiest places on earth.

Sources with experience in Saudi Arabia could recall no recent incidents reflecting dissatisfaction among the Saudi Shiites, despite their minority status.

"Everybody in Saudi Arabia has it pretty good, including the Shiites," said one observer. "But the Shiites are traditionally less a part of Saudi society than the Sunnis."

At the same time, he pointed out, Saudi Shites, like their brethren in other Middle Eastern countries such as Lebanon, increasingly have been aware of themselves as a distinct community since the Iranian revolution and Khomeini's proclamation of an Islamic republic.

Evidence of this sentiment among Shiites in nearby Bahrain and Iraq is known to be a subject of concern among some Saudi officials. Aside from the danger of internal unrest in these fellow Arab countries, the Saudis regard potential Shiite agitation as another source of instability around the already troubled gulf through which much of the world's oil must pass.

In addition, the antiroyalist direction taken by the Shiite revolution in Iran is a troubling precedent for the Saudi monarchy, which sits on one of the world's few remaining absolute thrones.

The chaos in Iran is viewed with abhorrence here. Saudi temperment generally dictates quiet persuasion as a code of political conduct. Moreover, Saudi leaders are said to fear that the ultimate outcome of Iran's turmoil, particularly if it should spread, should be a triumph for Soviet-influenced leftists.