Iranian army troops opened fire on a crowd of anti-shah protesters yesterday, reportedly killing five and sending thousands of enraged youths on a rampage through downtown Tehran. Scores of bank buildings were damaged, army vehicles were overturned and burned and one of the largest luxury hotels was stormed and damaged.

In addition to those reported killed in a fusillade of rifle fire at the downtown campus of the University of Tehran, scores of others were reported injured in the most serious confrontation since the Sept. 3 riot that left hundreds dead and ushered in material law in a dozen Iranian cities and towns.

The university campus crackled with automatic weapons fire as panicked youths fled in all directions, smashing windows of banks, setting fires and overturning vehicles on their way.

The International Hotel in the heart of the business district came under siege in midafternoon, as protesters set fire to two fire engines in front of the hotel, burned a bus and smashed hotel windows before they were dispersed.

It was the first time in the three-month wave of violence that rioters fanned out over such a wide area, and they left behind smashed windows in virtually every bank building in the central commercial district. Previously, demonstrations had been relatively localized and easily contained in this sprawing city of 4.5 million.

There were conflicting reports as to what touched off the shooting. So far military authorities have adopted policy of not meddling with the demonstrators, even if they marched in defiance of martial law, as long as they did not destory property along the way.

Organizers of the movement that has caused mounting civil unrest here and has threatened to topple the government suggested that Shah Mohammad Reza Pahlavi and his military advisers may abandon the policy of restraint as a result of yesterday's riot.

Meanwhile, officials of the Iranian National Front, which encompasses political groups ranging from moderate to Marxist and has a shadow cabinet prepared to step in as a provisional government, flatly rejected any possibility of negotiations with the shah for a compromise political solution.

The National Front, while untested in terms of popular support because it has been illegal, is the largest and most extensively organized opposition.

In an apparent reaction to an uncompromising stand taken Friday by exiled Shite Moslem leader Ayatollah Khomeini, National Front executive council member Darius Farouhar said in an interview that "under no conditions" will the opposition attempt to negotiate a political compromise as long as the shah is in power.

Earlier, former prime minister Ali Amini announced that National Front leader Karim Sanjabi, who met with Khomeini near Paris, would meet next week with the shah in an attempt to work out a political solution to the crisis, meaning the formation of a provisional government until Iranians are allowed to vote in the June elections promised by the monarch.

Khomeini, in turn; declared in Paris that if the National Front negotiated with the government it would be banished from the protest movement.

Farouhar said in the interview yesterday, "The National Front accepts what Khomeini has said. What the shah has done during the last 25 years, we can't have anything to talk with him about, and there is no guarantee that if the National Front talked with him he would respect what he said.

"For this reason, we have blocked any possibility for Dr. Sanjabi to discuss anything with the shah," he added.

When asked under what conditions the National Front would negotiate with the shah, Farouhar said, "If the shah surrenders completely for the prevention of chaos in Iran, the National Front will discuss surrenders."

In an unmistakable hardening of the group's position from several days ago, he added, "When Saigon fell, what happened? There were no discussions then."

The United States and other Western governments had been holding out hope that moderate opposition leaders not closely aligned with the conservative Moslem leaders who originated the anti-shah movement were coming closer to the idea of striking a compromise with the shah, resulting in a provisional constitutional monarchy.

Their hopes appeared dashed last night with Khomeini's hard-line stance and the National Front acceptance of the ultimatum of the Moslem leader, who is advocating an "Islamic socialist republic" to replace the monarchy.

Yesterday's riots added another dimension to the imponderables of the turbulent political situation here: If the riots continue to escalate and the shah orders a severe crackdown, the army could launch a campaign of repression after weeks of experimenting with a policy of relative appeasement.

If the shah fails to act decisively, the increasingly frustrated army generals could take matters into their own hands and seize power, either with or without the shah retaining his throne.

Western diplomats suggested last night that the determining factor could be whether today brings another wave of rampages through Tehran's streets.

Yesterday began peacefully at the university with several thousand students marching inside the fenced campus.They were shouting, "Khomeini, God preserve you, the American shah must be hanged," but there was no movement toward the street.

There are conflicting accounts of what happened next, but some witnesses said the students stormed through the iron gates toward a phalanx of soliders. Several protesters interviewed said there no such move, and that the troops outside fired through the gates and then chased the protesters into the campus.

As the rifle fire intensified, with a total of about 300 rounds being fired over a 20-minute period, hundreds of student could be seen running through a rear gate. Some of them collected trash and burned it in the streets, while others picked up small automobiles and positioned them in the street as a barricade.

The crowd then reformed several blocks from the campus and began a march through the central district, smashing windows of banks, liquor stores and movie theaters - all of which have been traditional targets of Moslem protestors.

At the Intercontinental Hotel, one of Tehran's largest, another group converged just as two dump trucks - apparently commandeered by the protesters - suddenly spilled their loads of sand and stone into the street, causing a massive traffic jam. Youth with wooden staves smashed the windows of a bus and fire engines and turned over three cars and set them afire.

The bus exploded in a flash and skidded across the street from the impact. The protesters turned over and set ablaze and effigy several blocks down the street and set fire to a night club, which was gutted.

The protestors seemed to attack the hotel as a matter of course while marching by, rather than selecting it as a specific target to demonstrate growing anti-Western feeling.