The United States yesterday firmly rejected Iranian demands for the extradition of deposed Shah Mohammad Reza Pahlavi and insisted on the safe release of the approximately 60 Americans held prisoner at the occupied U.S. embassy in Tehran.

But U.S. officials said privately there is no sign of a break in the impasse caused by the decision of Iran's ruling powers to condone making U.S. diplomats physical hostages to their demands.

The United States, the officials said, is facing a situation in which the Iranian government is unwilling or unable to honor the norms of international law and diplomacy.

That situation, the officials added, has created grave obstacles to solving both the immediate problem of securing the hostages' release and the longer-range question of future U.S. relations with a country whose oil wealth and strategic geographic position are potentially major factors in maintaining world peace and stability.

Publicly, the U.S. posture was to say as little as possible about the situation while trying to work behind the scenes to resolve it. Tight clamps were put on all public utterances by U.S. officials, and the White House, after a day of internal debate, finally canceled earlier-announced plans for a statement by President Carter.

But the administration did have Hodding Carter, the State Department's spokesman, enunciate the basic U.S. position. In response to questions about whether the shah, who is undergoing treatment in New York for cancer, would be returned to Iran, Carter gave a flat "no."

He also added: "We expect the government of Iran to secure the release of the Americans and to return the embassy compound to our control."

Beyond that, administration sources would say only that every effort is being made to appeal to the Iranian government's sense of reason, humanitarianism and internationally accepted standards of conduct. The sources said this effort is being pursued directly with the Iranian authorities and through the intermediary offices of other governments.

The matter is so sensitive, the sources said, that they refused to identify the other countries involved, except to say that some are Middle Eastern Moslem fundamentalist forces controlling Iran.

But, the sources conceded, the chief problem impeding the U.S. effort is that the titular authority in Iran, the government of Prime Minister Mehdi Bazargan, cannot control the mobs of demonstrators who took over the embassy Sunday and who give their allegiance to Ayatollah Khomeini, the religious leader who is the real power in Iran.

The result, the sources said, is a situation in which Khomeini and other religious leaders, apparently in an effort to deflect attention from internal problems, have been egging on the demonstrators despite Bazargan's reservations.

The sources said the Bazargan government appears to have succeeded, at least for the moment, in assuring the hostages' safety. Beyond that, the sources said, the Bazargan government is unwilling to move against the demonstrators holding the embassy -- a fact underscored yesterday by official statements from Tehran and at a news conference from Tehran and at a news conference here by Ali Agha, charge d'affaires of the Iranian embassy.

He said his government as "the servant of the people," stands behind the demands of the Tehran demonstrators that the shah be returned to stand trial in Iran and that his wealth in the United States be returned to the Iranian people.

Such actions, Agha asserted, would be a "positive sign" that the United States is repudiating what he called plotting against Iran engineeredd by what he described as such allies of the shah as former secretary of state Henry A. Kissinger and New York banker David Rockefeller.

Unless the demands are met, Agha warned, he could not say what might happen in terms of the hostages' safety or other possible steps, such as a cutoff of Iranian oil exports to the United States.

Although administration officials said oil does not appear to be a factor in the current dispute, they conceded that a cutoff of supplies from Iran could force a return to gasoline lines or, at the least, spot shortages of oil in the U.S. market.

Formerly the second-largest exporter in the Organization of Petroleum Exporting Countries, Iran now is sending an estimated 3.2 million to 3.8 million barrels a day of oil into the world market. Of that, 600,000 to 800,000 barrels a day -- roughly 10 percent of daily imports -- comes to the United States, whose principal foreign suppliers are Saudi Arabia, Nigeria and other Arab producers such as Libya.

At his news conference, Agha also charged that the United States had been warned in advance of what might happen if the shah were allowed to come here from his former refuge in Mexico. After the shah entered the United States, he said, Iran formally asked for his extradition. e

It was understood, however, that no formal request for extradition in the diplomatic or legal sense has been made to U.S. authorities. There is no extradition treaty between the two countries, and reliable sources said that if the shah wishes to remain here, he would have no trouble getting U.S. courts to block his forced deportation back to Iran.

Agha's charges about Kissinger and Rockefeller, both of whom are known to have argued for the shah's admission to this country as a long-time former U.S. ally, seemed likely to rekindle disputes about the administration's wisdom in allowing him to come here.

Reliable sources said David C. Newsom, undersecretary of state for political affairs, had taken the lead for months in turning back efforts to give the shah a U.S. visa out of concern for the repercussions that would be triggered in Iran.

However, the sources said, Newsom was overruled when the shah's cancer gave primacy to arguments about humanitarian considerations. That factor, the sources said, had been carefully and repeatedly explained to the Iranian authorities.

Officials at New York Hospital said yesterday that a cancerous lymph gland tumor in the Shah's neck has enlarged since his recent surgery and another stone has been discovered in a bile duct. The new stone may require surgery, the hospital said in a statement.

"Although the shah is recovering from the immediate effects of surgery the situation remains serious in view of the underlying malignancy and the continuing biliary tract disease," the hospital said.