The United States escalated its economic and diplomatic pressure on Iran yesterday by freezing all Iranian govenment assets in this country and successfully blocking Iran's call for a United Nations Security Council debate on the bitter dispute between the two countries.

As the war of nerves over the fates of the American hostages being held in Tehran entered its 11th day, President Carter signed the freeze order early yesterday morning shortly after receiving reports that Iran planned to withdraw its assets from U.S. jurisdiction. Administration officials estimated that the order will affect about $5 billion in Iranian government assets, including deposits in the overseas branches of American banks.

Meanwhile, Secretary of State Cyrus R. Vance flew to New York yesterday to assume personal supervision of the U.S. effort to block any Security Council debate about the Iranian situation until after the hostages are freed.

Vance's diplomatic initiative succeeded when the U.N. Security Council president, with the unanimous backing of the other 14 council members, informed Iran that there will be no discussion of Iran's complaints against the United States as long as the hostages are bing held. [Details on Page A28.]

The two moves appeared to be part of an administration strategy gradually to increase economic and other pressure on Iran while keeping international attention focused on the some 100 hostages -- including 60 to 65 Americans -- who are being held by Iranian students inside the U.S. Embassy in Tehran.

Two special U.S. envoys who have been trying since last week to meet with Iranian officials were recalled to Washington yesterday, the State Department said last night.

Former attorney general Ramsey Clark and Senate aide William G. Miller were brought home from Istanbul, where their mission stalled when Iranian leaders refused to talk to them.

White House and other administration officials asserted that the freeze imposed on Iranian government assets was not a form of retaliation for the taking of the hostages, but a U.S. response to Iran's plan to withdraw the assets.

In announcing their intentions, Iranian authorities said about $12 billion in withdrawals was involved. Administration officials estimated the extend of Iranian government assets in U.S. jurisidiction at less than half that amount. Whatever the correct amount, the American countermove represented a new dimension in the confronation that began with the Nov. 4 takeover of the embassy in Tehran.

Treasury Secretary G. William Miller informed Carter about Iran's intentions shortly before 6 a.m. yesterday. Two hours later, Carter signed an executive order declaring that there exists "an extraordinary threat to the national security and foreign policy of the United States" and imposing the freeze.

Carter acted under his authority in the International Emergency Economic Powers Act. Treasury Department officials noted that the act empowers the president to take additional steps, including a full-scale trade embargo and the freezing of privately owned Iranian assets in the United States.

In announcing the freeze order, the White House issued a brief statement, saying the step was being taken "to insure that claims on Iran by the United States and its citizens are provided for in an orderly manner."

The order affects the bank deposits and other property of the government of Iran, the Iranian national bank and such government controlled entities as the Iranian national bank and the Iranian national airline, Iran Air. (Miller said last night that Iran Air would be given a license to depart from the United States.) The freeze does not affect the private holdings of individual Iranians, including Shah Mohammad Reza Pahlavi, who is undergoing treatment for cancer in New York.

[NBC News reported last night that the shah plans to leave New York and return to Mexico within 10 days. However, State Department officials say they know of no imminent plans for the shah's departure and added that according to their information, he would not be in condition to leave for weeks.]

The United States did not seize Iranian property. However, under the freeze order, the transfer or use of any of the assets will require approval by the Treasury Department, placing them effectively under the control of the U.S. government.

Briefing reporters yesterday, Treasury Secretary Miller said the largest Iranian government bank deposits are about $1.3 billion in Treasury notes held by the Federal Reserve Bank of New York.

He said the order should not affect commercial transactions between private firms in two countries or the purchase of food and other commodities by the Iranian government, which, he noted, has "substantial resources" outside the frozen U.S. accounts.

Miller said other governments were informed of Carter's action yesterday and supported it. Officials in Saudi Arabia, he said, "understood the action and were sympathetic to it."

The Treasury secretary also down-played the possibility of Iranian retaliation against U.S. assets in Iran, saying these are "not large."

Although the freezing of Iranian assets was the most dramatic development yesterday, the U.S. shift toward a harder line actually was signaled most clearly by the drumbeat of tough, no-compromise language with which the administration announced its determination to block Iran's bid for a Security Council meeting.

Calling the Iranian move an attempt to put the United States "on trial before the Security Council," State Department spokesman Hodding Carter said: "There is no way this government can or will negotiate under the gun of its own people being held illegally in its own compound in Iran."

Again and again, the spokesman reiterated that the United States is determined to prevent any discussion that might deflect world attention from what the administration contends is the central issue in the dispute -- the plight of the hostages.

Although he stopped short of saying so explicitly, the net effect of his message was that the hostages must be freed before any other aspects of the crisis are discussed. Once that is done, the spokesman made clear, the United States then will be willing, in the Security Council or any other appropriate forum, to discuss the complaints and issues that Iran want to raise.

Administration officials said the tactical shift unveiled yesterday had been decided upon by the White House and reflected the feeling of Carter's top advisers that the president must make unmistakably clear his resolve not to yield on matters of basic principle.

The White House, they added, considers that an imperative, in terms both of keeping domestic American outrage from boiling over into demands for drastic action and underscoring to the Iranians that the United States will not back down.

In this respect, the sources said, the administration feels its strongest weapon is the increasing isolation in which Iran has found itself as the result of its actions

They cited yesterday's blocking of Iran's attempt to get a Security Council hearing as evidence that the opinion of the world community is overwhelmingly on the U.S. side and added that once Iran realizes it has no significant support for its present course, it is more likely to become more reasonable about the hostages.

Such reasoning appeared to represent a defeat for some administration officials, particularly within the State Department, who are known to have argued for a more flexible stance. As recently as Tuesday, some of these officials had professed to find possible grounds for accommodation in the Iranian proposals that included the call for a Security Council meeting.