The difficult and dangerous game of transitional politics now being played in Iran is forcing the Carter administration to put its money publicly into a bet that many U.S. specialists think cannot possibly be won.

That bet is that Shahpour Bakhtiar, named by the shah as the prime minister who would run the country while the Iranian monarch goes abroad, can form a viable government and end the year-long violent protest that has shattered the shah's authority.

The creation of a vacuum where the shah ruled in dictatorial fashion is also forcing a reassessment by the administration of its previously hostile attitude toward Ayatollah Ruhollah Khomeini, the exiled Moslem leader who has become the leading symbol of opposition to the shah.

This grudging and highly tentative reassessment of Khomeini is a measure of the alternatives many administration experts see being posed in a building power struggle to determine who will succeed the shah once Bakhtiar's role is played out.

What the White House appears to fear most is not so much a military coup that would succeed and end the disruption of Iranian life and oil exports, but an attempted coup that would fail because troops would not carry out orders to repress demonstrations.

That would plunge the country into chaos and trigger a potential, still undefined reaction by the Soviet Union that would in turn force the administration to take action. That appears to be President Carter's ultimate nightmare on Iran.

Against this backdrop, Khomeini appears to be quickly evolving in the administration's view. After being seen as a primitive Islamic fascist not to be dealt with, he is becoming a political figure with whom it may eventually be possible -- and necessary -- to make a deal.

The ayatollah's new moderation in public statements about the shah and Iran's future relations with the United States have caused the administration to tone down its public statements about him, and to say publicly that U.S. officials are in contact with Khomeini's aides on some kind of regular basis. U.S. officials still have not asked for direct contacts with Khomeini, however.

Asked at yesterday's State Department briefing about Khomeini's new tone in interviews given from his suburban Paris refuge, spokesman Hodding Carter declined to comment directly.

But he then said, "We are certainly pleased by any remarks by any Iranian party that help calm things." U.S. officials said the remark was intended as encouragement to Khomeini to continue speaking moderately.

"The fact is that we didn't know what Khomeini thought or represented when we were dealing with that phenomenon as the dark side of Islamic fanaticism, and we don't have any better idea today from these new statements who he is," one U.S. official conceded. "We just have to try harder to figure out what he wants, an idea that was unthinkable for our policymakers even a month ago."

At the briefing, Hodding Carter also dealt with the grim alternative of a military coup by delivering the State Department's strongest appeal to the Iranian military to give Bakhtiar's government a chance.

"We have urged the military to give its full support to the new government. We do not believe a military takeover would resolve Iran's problems," Carter said, adding under questioning that the State Department had received reports of a potential coup.

Bakhtiar's readiness to carry out the shah's wishes in forming a cabinet has killed his chances to win support from the political opposition and he has no backing in the armed forces, U.S. analysts believe.

"You can't form a government until you have decided who is going to run the country, and that has not yet been decided in Iran," said one official. "We can only hope Bakhtiar can buy time to get things sorted out, but that hope dies a little each day."

For some U.S. analysts, Prime Minister Bakhtiar is the Iranian equivalent of Alexander Kerensky, who headed the provisional government that deposed the czar of Russia in February 1917 only to be deposed by the Bolsheviks in October. But analysts who use the Kerensky analogy admit that they see no Nikolai Lenin on the horizon, from either right or left.

It is far from clear that Khomeini would accept a direct role in a political solution. He may see his role as that of an Islamic visionary able to set boundaries for and exercise veto control over a functioning post-shah regime.

"Maybe he doesn't want to be the king; maybe he wants to be the pope," said one official. "Perhaps he just wants to return and be what the shah would not let him be -- the leading Islamic figure in Iran."

Some analysts suspect that turmoil will continue until eventually the military emerges as the only organized force left in the country and openly takes control to protect its privileges.

With an edge of cynicism, some foreign diplomats here note that by backing Bakhtiar so strenously in public, the Carter administration has now established that it did everything possible to prevent a coup. That record might be important in trying to establish to the Soviet Union that Washington had nothing to do with a military takeover.

Whether Moscow would believe that, some U.S. and foreign diplomats admit, is still an unanswered question.