It was a bizarre tableau: Iran's prime minister sitting back, tieless and stubble-bearded, with his bare foot propped up on an elegant conference table against the background of the U.N. symbol on pleated, powder-blue draperies.

As Mohammed Ali Rajai pointed at the sole of his foot sticking up there for the cameras, a few scarcely repressed snickers floated across the faces of U.N. correspondents more accustomed to coverng the polite world of pin-stripe diplomacy. But for those who chose to look more closely, Rajai's performance was an introdcution to a new world -- Iranian diplomacy -- and a dramatic illustration of the chasm separating Iran's leadership from the American and international establishments trying to arrange for release of 52 American hostages held in Iran for nearly a year.

The chasm was particularly evident last week as U.S. policymakers saw in news of Rajai's plans to visit the United Nations an opportunity for progress in their efforts to get the hostages released. First President Carter and Secretary of State Edmund Muskie made public offers to meet with him, mindful of Iran's need for military help and the stores of already purchased U.S. equipment frozen in the United States because of the hostages.

But then, as it became increasingly obvious that Rajai had nothing of the sort in mind, U.S. diplomats toned down expectations and said their main hope centered on the possibility that Rajai would realize how the hostage onus has isolated Iran at a moment when it needs help for its war with Iraq.

As it turned out, however, Rajai had left Iran for the first time in his life, but Iran and its radical Islamic perspective had not left him. Without budging from his side of the chasm, he refused to meet any Americans on the hostage issue, proclaiming that Iraq's attack on Iran was actually part of a U.S.-led scheme against the Iranian revolution. As for isolation, Rajai declared confidently that while Iran may encounter opposition from established governments and their well-tailored representatives here, the "freedom-loving peoples of the world" are solidly behind it.

Once again, U.S. and U.N. officials seemed to have made plans on the basis of what looked logical in Washington and New York, while Iranians followed their own course along the other side of the chasm.From the day the hostages were seized, Nov. 4, 1979, the gap has seemed just as unbridgeable for U.S. and U.N. diplomats accustomed to the traditional rules for relations between governments. Time and again, it has spread to become an uncrossable obstacle as Iranian leaders, in particular Ayatollah Ruhollah Khomeini, acted in the light of their own, vastly different set of rules.

In the revolutionary atmosphere of Tehran brought for two days last week to the United Nations, international conventions for the protection of diplomats -- repeatedly cited by the United States and other nations as reasons that Iran should release its captives -- appear to mean little more than the conventions requiring prime ministers addressing the U.N. Security Council to shave, put on a necktie and keep their shoes on.

So on one level, Rajai's appearance was the uniform of Tehran's lay leaders, as was the olive drab fatigue jacket of his unidentifed constant companion and adviser. On another level, however, the attire also was a declaration that the rules in Washington and the United Nations -- be they sartorial or diplomatic -- do not apply to Iran's Islamic revolution.

In the perspective of the 46-year-old former mathematics teacher whose Moslem fundamentalism grew out of years in Tehran slums, they are the conventions of a Western, largely Christian tradition, precisely the sort the late shah tried to force-feed into Iran and that Khomeini led his Islamic revolution to get rid of. Moreover, Rajai asked, where were those who worry about these conventions when the shah's secret police was torturing Iranian dissidents, himself included?

Rajai's very trip to the United Nations after a long boycott of its proceedings was in some respects an acknowledgment that even with its revolution, Iran has to deal with other nations. The U.S. ambassador here, Donald McHenry, called it a "first step."

But other first steps have in the past led only to stumbles, in part because U.S. or other officials trying to be helpful have worked on the assumption that the first step would lead to subsequent ones along the familiar, if twisted, path of diplomcay. Iran has chosen instead to follow its own paths.

For example, when the United States thought it had come close to a deal for release of the hostages last March, U.S. officials were working through Abol Hassan Bani-Sadr on the hope that, as president, he was speaking for the nation. But in the peculiar world of Islamic revolution, the only one who could really speak for the nation was the Imam Khomeini, and he allowed the agreement to fall through.

Similarly, U.N. Secretary General Kurt Waldhiem's mission to Tehran last January led to nothing when the Iranian leadership used his presence there to dramatize the horrors of repression under the shah rather than carry out the diplomatic agreement U.S. and U.N. officials thought they had carefully constructed. Members of Waldheim's team, including Moslems, came away shaking their heads because, they said, the Iranians were playing by rules of their own.