Negotiations were reported under way here today between representatives of the armed forces and of exiled Ayatollah Ruhollah Khomeini to avoid a showdown that could destroy the last vestiges of state authority.

Disclosure of the talks -- contradicted by Khomeini's continued denials that any were taking place -- came as the Moslem leader announced from his exile base in Paris that he expects to return to Iran shortly.

"With the help of God, I will be among you after a few days," Khomeini said in a message to his millions of followers here.

Meanwhile, antigovernment demonstrators shouting "Hail to Khomeini" marched peacefully in Tehran again today, and at least 12 deaths were reported in protests elsewhere as the government of Prime Minister Shahpour Bakhtiar struggled to assert control.

Despite Khomeini's denial of contacts with Iranian military representatives, informed sources here indicated the talks started two days ago. They provided no insight into their progress or possible duration.

Analysts suggested, however, that if Khomeini's promise to end his 14-year exile within a few days was accurate, the negotiations must be nearing a successful conclusion, if they have not already done so.

At stake was winning senior generals' agreement for the return of Khomeini, the shah's archenemy and considered by many military men as a traitor, and his key demands for abolishing the monarchy and establishing an Islamic republic.

Diplomats, who said the talks were the only reasonable way out," described the situation as "very delicate."

They noted that the shah had to argue long and hard before the senior generals agreed reluctantly to accept Bakhtiar in a face-saving formula that allowed the sovereign to leave Tuesday on a vacation with the outward appearances of the monarchy intact, instead of abdicating.

In recent days, Bakhtiar has predicted that if his government fails, the armed forces would stage a coup rather than accept Khomeini's return and takeover of power.

It was not clear whether Bakhtiar was involved directly in the talks or was simply being kept informed.

"The zillion-dollar question," one diplomat said, "is how large a psychological step did the senior generals take Tuesday when the shah left. Perhaps there's only another half-step for them to take to accept a republic -- or perhaps not."

The identity of the negotiators for both sides was not known. However, speculation centered on Mehdi Bazargan, a much respected opposition leader, as the most likely representative of Khomeini.

Bazargan was unavailable for comment. However, analysts said Bazargan's only partially successful effort over the past three weeks to persuade militant leftist oil and railroad strikers to return to work made him an intermediary the generals would respect, if not necessarily like.

As conservative as the generals themselves, Bazargan could be expected to argue that every passing day favors extremists in their paralysis of the state administration and economy.

The United States has reportedly thrown its support behind Bazargan, but not openly, since Washington is said to be learning from its past errors of publicly backing Iranian politicians and thus compromising whatever chances they may have had.

Another possible argument centers on a common fear of local Communists, leftists and the Soviet Union.

Lt. Gen. Robert Huyser, the U.S. Air Force general who arrived here more than two weeks ago, was reported still meeting with Iranian counterparts, arguing that moderation, not a coup, was needed to forestall a Communist takeover here.

It is unclear how effective such arguments may prove.

Even those claiming to know the senior generals well are reluctant to speculate about their control of draftees and junior officers under their command, who appear sorely tried by Khomeini's calls for fraternization with his followers.

But in recent weeks, the military commanders have been less given to claiming that they could set things right in short order. Their caution reflects the growing possibility of a fatal split in Iran's armed forces.

The senior generals had little success during the November-December military government in reestablishing order, much less getting strikers back to work.

Moreover, unconfirmed opposition reports of growing numbers of desertions and mutinies on military bases may be true.

Nevertheless, the prospect of having to accept Khomeini's return to a position of power here after 14 years of exile still adds up to "something of a shock wave," in the words of one diplomat.

Khomeini's National Front opposition allies now claim the threat of a military coup has faded. They insist air force officers have told their commanders they would not support any such efforts and they claim that the ground forces, half of whom are draftees, are very demoralized.

No matter how tactfully any compromise might be worded, Khomeini is so clearly now in the driver's seat that, the most the military commanders could hope for is a polite transition before the monarchy is dismantled.

Whatever hopes Bakhtiar and the army might have had for preserving the monarchy seemed to evaporate when the process of installing the government dragged on too long.

Instead of gaining credit for finding the face-saving formula for persuading the army to allow the shah to leave the country, Bakhtiar is seen increasingly as a powerless figure.

Making the generals' decision even more difficult is the knowledge that under a republic the special status enjoyed by the armed forces with the Pahlavis over the past half century would come to an end.

Analysts argue that the more realistic commanders realize that the Iranian economy is in such serious disarray that the armed forces cannot hope to receive their past share of the budget, usually 24 percent or more.

The handwriting on the wall is for a smaller Iranian military establishment. It is also considered likely that Iran's new masters will purge an officer corps so closely identified with the Pahlavi dynasty.