The invasion of the Great Mosque at Mecca, ended yesterday after two embarrassing weeks, has dealt a heavy blow to Saudi Arabian prestige and raises troubling questions about security in the kingdom.

Doubts generated by the mosque takeover have been amplified in the restive atmosphere created throughout the Middle East by the Islamic revolution in nearby Iran. They also have been exacerbated by the Saudi royal family's attempt to diminish the scope of what, in light of accumulating reports, has emerged as a major uprising by well-organized rebels supplied with modern weapons.

As a result diplomats and anaylsts in Saudi Arabia and other Arab countries have begun to reconsider the widely shared impression that Saudi Arabia was immune to Iran's call for Islamic agitation and that the royal family was safe from any challenge to its rule.

U.S. diplomats in Riyadh, the Sauid capital, met among themselves last week to discuss the safety of Americans citizens here in view of attacks on U.S. installations elsewhere, a concern that would have been laughable only a short while ago.

The mosque operation, however, demonstrated that a determined group can get around Saudi security precautions well enough to plot a spectacular attack, procure heavy weapons and ammunition and infiltrate one of the kingdom's biggest cities with hundreds of armed men.

In addition, it showed that despite the kingdom's vast wealth there is a pool of dissatified youths ready to sacrifice their lives in the name of Islamic reform and opposition to the royal family's way of governing.

Reports circulating within Saudi Arabia said rebels struck unsuccessfully at Medina and Taif on the same day the Mecca mosque was invaded, leading to fears among some Saudi officials that a coordinated uprising was in progress and that ministries in Riyadh were about to be attacked.

Saudi security forces also made a number of arrests in Jeddah after the first Mecca invaders were captured and interrogated, these reports said, indicating the organization indeed had a broader base than the "corrupt gang of Islamic renegrades" blamed by Saudi officials.

Information Minister Mohammed Abdo Yamani denied the reports. But longtime residents of Riyadh said their friends among the lower ranks of the royal family were so shaken by them that they refused evening invitations and remained inside guarded houses. One Saudi official in Jeddah agreed to attend a European embassy reception only on condition that he be accompanied by a National Guard detachment.

Since then, there have been reports in Beirut of disturbances as well in the eastern province, home of most of Saudi Arabia's approximately 120,000 Shiite Moslems. Authorities dispatched an extra 20,000 troops to handle the mobs late last week, according to the reports citing Saudi sources and returning travelers.

The Mecca attack itself, according to accounts now available, was mounted on a larger scale, with bigger weapons and a stronger rebel force than Saudi authorities earlier admitted. The "band of Moslem renegades" cited in the first Interior Ministry communique comprised, by the ministry's most recent disclosures, several hundred gunmen.

Diplomatic sources in Riyadh said they were armed with .50-caliber machine guns. This level of weaponry, still not reported by Saudi authorities, helps explain why the invaders were able to hold out for two weeks and inflict casualties among Saudi troops estimated by Prince Naif, the interior minister, at 60 killed and 200 wounded.

Maj. Mohanned Zuweid Nefal, a Saudi special forces commander who fought the rebels, told a Saudi television audience that at one point the gunmen controlled several minarets in the mosque overlooking Mecca and its surroundings and were able to fire "as far as the mountain heights where some innocent people fell victim to their sniper fire."

The scope of the fighting indicates considerate organization, the diplomatic sources said, pointing out that the heavy weapons probably would have to be stolen from Saudi military stores or smuggled into the country, then sneaked into Mecca broken down for reassembly inside the mosque. Another sign of organization, they added, was the ample supply of food and ammunition carted into the mosque, apparently through underground passages, in anticipation of the takeover.

The gunmen's motives remain unclear more than two weeks after they began their attack The Saudi government has portrayed them in Islamic deviationists whose mission was proclamation of a mahdi, or the final prophet, awaited by a small strain of Saudi Moslems.

The man the gunmen hailed as the mahdi at the outset of their occupation Nov. 20, identified as Mohanned ibn-Abdullah Qahtani, was killed in the final cleanup fighting in flooded passageways underneath the 38-acre mosque, Naif said.

Other sources, however, said the attackers also demanded a purification of Saudi society, ridding it of such Western innovations as television, organized soccer and women in business. This gave their occupation of the mosque a political character as well, dramatizing debate in the kingdom over the swift pace of oil-fueled modernization.