THE FAILURE of last week's attempt to rescue the American hostages in Tehran makes even more likely the grim possibility, already acknowledged by President Carter, that they may reamain in captivity until after the November elections, or even into the indefinite future.

It is a possibility that no American could have anticipated last November, when the Teheran embassy was seized. What few Americans understand is that holding hostages as an act of state was long an established tradition in the Islamic world, and it is to the restoration of Islamic traditions that the Ayatollah Ruhollah Khomeini has appealed repeatedly in his attempts to hold his country together. He may need to hold the hostages -- or even order the taking of new ones -- until his Islamic republic is safely established. And that could mean a very long time.

The anatollah, is unlikely to stay around for an eternity -- although his ghost could haunt Iran for years to come. Thus, America's problem, to Carter's undoubted chagrin, has become intermixed with the problem of Khomeini's succession.

As long as Khomeini is around, the Islamic Revolution will probably manage to continue blundering its way to survival despite all the internecine feuding, administrative chaos, diplomatic isolation and economic sanctions. But after that, the contest would be wide open. An Iran is, after all, a prize worth fighting for.

For centuries, two institutions had kept Iran united. One, the monarchy, was lost with the overthrow of the shah. The other, still intact, is the Shiite mosque. Just as the shah paved the way for the destruction of Iran's monarchy by involving himself in every detail of government, from the cultivation of beans to the purchase of sophisticated weapons, so the leaders of the Shiite mosque -- the mullahs -- today are determined to become Iran's rulers.

Following in the footsteps of the monarch they never stop cursing, the mullahs, organized in the Islamic Republic Party, believe that their ambitions should be underwritten by the United States. They love America no more than the shah did. But they understand that in Iran's present state of weakness, the superpowers must have even more influence than before.

They see the Soviet Union, in the long run, as the natural sponsor of the totalitarian Left. So they have only the United States to turn to as a counterbalancing force. A failure in signaling, together with tactical errors on both sides, kept Washington from receiving their urgent appeals to enter into a political dialogue. And then the attack on the embassy made things even more complicated.

Khomeini's public tirades have left Americans convinced that he and his followers are fanatically against the United States. In fact, they are against every influence of the contemporary world that might distract the faithful from their devotion to God. Khomeini has referred to the United States as a great taghoot , and this has been translated here as "the great Satan." But this is a mistranslation, for taghoot in the Koran refers to the idols that were destroyed by the prophet Abraham, and when Khomeini applies it to the United States he is not inciting Iranians to hatred but is calling on them to reject a materialistic, contemporary view of life.

Even the holding of hostages is not as anti-American as it is traditionally Iranian. For centuries, the exchange of hostages was an integral part of the diplomacy of the East. Scores of Persian and Byzantine princes lived their lives as hostages, serving as guarantors of a brittle peace. Within ancient Persia, every major feudal cheif would send one of his sons as a hostage to the king's court, where he could safely escape the ravages of war. The advent of Islam did not change this tradition, but in fact gave the religious blessing it had lacked.

Against such a background, it would have taken some doing to persuade Khomeini that he should reject the taking of hostages because it is anathema to modern -- that is, western -- diplomacy. Instead, Khomeini was persuaded of two things: first, that the United States wants to keep him from realizing his dream of an Islamic republic, and, second, that the largest crowds in Iran approve the holding of the American hostages.

Although he did not plan the embassy seizure and was not even consulted by those who pulled it off, he saw immediately that by supporting their coup he could offer something to please the crowds, who have little else to march for in Iran these days. And he reasoned that as long as the hostages were held, the United States would be unable to plot with a free hand against his Islamic republic.

What,then, can be done to change Khomeini's mind about the hostages? He has said repeatedly that he will go where the crowds go, and in this he shows his political astuteness, for he is a creature of the crowds. Accordingly, two things should be done:

First, the ayatollah must be persuaded that the United States does not intend to remain a hostage of the hostage situation forever, and that a continued stalemate could lead to all the harm he fears. Second, Khomeini must be made to see that the crowd backing release of the hostages is larger than that still endorsing their captivity.

President Abol Hassan Bani-Sadr has tried his hand in both fields. The results have been disastrous for everyone. Instead of convinving the ayatollah that making a deal with Washington would strengthen the nascent republic, he has given the impression that he was succumbing to the charms of "The Grand Idol." Khomeini has dismissed his first prime minister, Mehdi Bazargan, for precisely the same reasons. As for out-mobbing the mobsters at the embassy, Bani-Sadr so far has failed to produce "the great masses of the believers" he has been promising in his daily editiorials in his own newspaper.

Who, then, could win the "biggest and most violent crowd" competition that remains the center of Iranian political life today? The mullahs could. They could succeed where Bani-Sadr has failed. And they could propose a settlement with Washington without being suspected by Khomeini of "succumbing to the idol."

The mullahs have been sending signals to that effect to Washington for a long time. At the end of 1978, when the United States finally decided not to back the shah, the mullahs organized a beauty parade of their own in order to attract Washington's attention. But the United States put its chips on Shahpour Bakhtiar before ditching him in favor of Bazargan. The mullahs destroyed Bazargan just as they had previously forced Bakhtiar out. But the United States chose to bank on Bani-Sadr, to the chagrin of the mullahs.

That chagrin is understandable. The mullahs, unlike Bakhtiar, Bazargan and Bani-Sadr, are known to be dedicated anti-Communists and Russia-haters. Some of their leaders almost certainly cooperated with the shah's Savak police in the name of "the common fight against communism." When the hostages were seized, most of the leading political mullahs condemned the move and demanded their release. During the country's elections the mullahs were in the forefront of the attack on the Left.

When the Soviet army invaded Afghanistan, the mullahs reacted with more vigor than Carter himself, forcing the government to attend the Islamic meeting on Afghanistan in Islamabad despite opposition from almost all other groups in Iran, including the "students" at the embassy. Last month, it was again the mullahs, who dominate the Revolutionary Council, who vetoed a foreign ministry decision that Iran should attend the Moscow Olympics.

And a week ago the mullahs launched the most massive anti-Left witch-hunt in Iranian history, ordering their crowds onto the campuses of the country's nearly 300 colleges and universities to purge them from leftist elements. Yet this is no more than a foretaste of the real bloodbath that is sure to follow if Iran continues without an effective government.

The mullahs today represent the only group capable of coming out on top in an almost Darwinian struggle for power. Led by Ayatollahs Mohammed Beheshti and Hashemi Rafsanjani, with Khomeini's heir-apparent, Allatollah Hossein-Ali Montazeri, acting as de facto party boss, the power-loving mullahs are prepared for a deal with Washington, with the fate of the hostages as just one element in the package.

Now fighting to turn the already mauled Bani-Adr into a tattered ceremonial figure, the mullahs mean to appoint a prime minister of their own choosing to maintain a direct hold on the government. With a majority in the new Islamic parliament which is to be formed in a couple of weeks, they are certain to succeed.

But a gevernment of the clergy -- a mullarchy, so to speak -- is unlikely to survive Khomeini. Mob politics is effective only in the short run. In the end, a political force must have a socio-economically identifiable constituency of its own to stand a chance of governing a fairly sophisticated society.

The core of the Iranian cricis is the absence of an effective government. And this is precisely why Carter's sanctions, short of a full blocade, will have no effect: There is no government to feel these pressures. Even if what now passes for a government chose to release the hostages, as in fact it has announced its desire to do, it would make no impression on the "students."

Who can give Iran an effective government? The United States has traditionally sought the answer to this question in individuals: the shah, Bakhtiar, Bazargan, Bani-Sadr and soon, perhaps, Beheshti. The results are already clear.

Seen from the longer-term point of view, and beyond Khomeini, only two forces would be able to give Iran a government: the middle class and the Left. The former fought the shah for political rights but cooperated with him on the economic front. It financed the Khomeini movement and played a key role in bringing down the shah, if only by abandoning him. But the middle class is now disillusioned with the mullarchy and is beginning to seek some means of entering the game to its own benefit.

The Left is still isolated, although strategically placed, and would need several more years before it could unite and begin to appear as an alternative to the masses.

Both the United States and the Soviet Union have a vital stake in Iran's future. Moscow has consistently backed Khomeini, hoping that he would stay around long enough to eliminate western influence from Iran. The United States, on the other hand, is immobilized by the hostage issue and its domestic implications in an election year. But Iran was important to the West before the embassy incident and would remain so long after the hostages are safely home.

There are people in Iran who are beginning to think beyond that eventuality and about the future government of their country. They understand the present mood of anger in the United States but believe that nothing durable can be achieved through anger -- nothing, perhaps, other than future remorse.

Despite the theatricals around the Tehran embassy and the indiscriminiate "punishments" ordered by Carter, there is still no deep-rooted anti-Americanism in Iran. During the past few months alone, and ever since the hostages were seized, more than a dozen Americans have been murdered or wounded in politically motivated assassination attempts in several countries, including some still tightly aligned with the United States. Nothing of this kind has happened in Iran. And those demonstrations around the occupied embassy in Tehran would not be there if their stage manager did not want them there.

To portray Iranians as rabid America-haters and then to react accordingly is a roundabout way of giving the ayatollah the excuse he thought he already had for his me-too support of the "students." As Sadegh Ghotbzadeh, Bani-Sadr's shifty foreign minister, recently muses, "Who's kidding whom?"