The United States, assured of backing from its European allies, is expected shortly to ask the United Nations Security Council to impose some form of economic sanctions against Iran because of its refusal to free the 50 Americans held hostage in Tehran.

Diplomatic sources said tonight that the plan to seek sunctions was worked out by Secretary of State Cyrus R. Vance during his four days of talks this week with the leaders of Britian, West Germany, France and Italy and the foreign ministers of the other 10 NATO member countries.

Still to be decided, the sources said, is whether the United States will ask for full or partial economic sanctions. The expection is, however, that any call for U.N. members not to trade with Iran will exempt, on humanitarian grounds, such items as food and pharmaceutical products.

If the Security Council refuses to apply sanctions, the sources said. Vance has been assured that the four major Western European allies and most other NATO members will cooperate with the United States in instituting an informal trade and financial embargo to keep the pressure on Iran.

[Speaking with reporters in Washington Thursday morning, Britian's ambassador to the United States, Sir Nicholas Henderson, painted a somewhat less optimistic picture of how the Europeans would view a call for economic sanctions. Such action, Henderson said, would have to be "fairly universal" to work.]

[The European, he said, would "would have to think about the consequences" of economic action, ncluding "the future of relations with Iran, and Iran's future relations with the West." While the Europeans have unanimously condemned the holding of the hostages, and given the United States full diplomatic support, Henderson said a request for sanctions "could lead to discussions that could open up what is now a unanimity of view."]

America's Western European allies have not been particularly enthusiastic about the idea of sanctions because they fear it could disrupt international trade and banking systems. In addition, many European countries import much of their oil from Iran and elsewhere in the Middle East.

Their concerns apparently were outweighed, however, by sympathy with the U.S. position, by a desire to prevent the U.S.-Iranian confrontation from becoming violent and by a desire to keep the United States on its path of trying to resolve the crisis through peaceful means.

Although NATO as an organization did not deal directly with the Iranian crisis during this week's talks, the foreign ministers of the 15 NATO contries signed a statement today condemning the seizure of the American hostages and declaring Iran "in flagrant violation of international law and human rights."

A senior U.S. official cautioned tonight that a formal decision to go to the United Nations to ask for imposition of sanctions will not be made until after Vance returns to Washington Friday and confers with President Carter.

European sources said, however, that unless Iran quickly complies with the Security Council's call for release of the hostages, it is "better than 90 percent certain" than the United States will ask for sanctions, probably as early as next week.

According to the sources, the Carter administration is waiting for two things before acting: An opinion by the World Court on the U.S. complaint that Iran violated international law taking diplomat's hostage and word from the Soviet Union about whether it will veto a sanctions resolution in the Security Council.

The court is expected to announce its decision Friday or early next week. The Soviets, who already have been asked bly the United States to clarify their positions on sanctions, are expected to reply within the next few days.

Washington is counting on the Soviet leadership's realization that a veto would almost certainly doom any chance of Senate ratifications of the U.S. Soviet Strategic Arms Limitation Treaty.

China, the other communist power with veto rights in the Security Council, is not expected to use it against a sanctions resolution.

A sanctions request would be based on Chapter Seven of the U.N. Charter which authorizes the Security Council to take measures against any country whose actions pose a threat to world peace. The United States and the allies will argue that Iran, though its defiance of the Security Council resolution on the hostages, has created such a threat.

U.S. sources said they could not predict whether sanctions would have any effect on forcing Ayatollah Ruhollah Khemoini's revolutionary government to release the hostages. Khomeini has said he will not recognize the United Nation's right to intervene in the crisis and has insisted that the hostages will not be released until the United States returns deposed Shah Mohammad Reza Pahlavi to Iran to stand trial.

The economic measures already instituted by the United States -- principally the freezing of more than 68 billion in Iranian assets held by U.S. banks and their foreign subsidiaries -- already have had severe effects on the Iranian economy by making it extremely difficult for Iran to obtain essential imports, according to a senior State Department official traveling with Vance.

The Carter administration wants to keep the pressure on by preventing Iran from finding other sources of goods and financing. That was the purpose of Vance's trip to Europe this week for talks in London, Paris, Bonn and Rome and at the NATO winter ministerial meeting here.

Although the sources would not be specific, they said that if the United States fails to get sanctions imposed by the United Nations, informal cooperation between Washington and its principal European partners would prevent Iran from obtaining imports and the financing to pay for them.

Details of such cooperations would be worked out later, but Vance and the European leaders are understood to have discussed a broad range of possible measures including future sharing of oil resources if Iran retailates by cutting off its oil exports to the West.

The Security Council now has full sanctions in effect against Rhodesia and partial sanctions involving supplies of military equipment against South Africa. Both have had mixed results but have not had noticeable serious effects.

Following tentative agreement by all parties in Rhodesia to reach a peace settlement, Britain has announced that it will lift its sanctions against Salisbury without waiting for the Security Council to do so. The United States has not announced a decision yet on lifting the sanctions, but is widely expected to follow Britain's lead.