The Security Council approved today, by a vote of 11 to 0, a U.S.-sponsored resolution calling for U.N. economic sanctions against Iran if Secretary General Kurt Waldheim fails to obtain the release of the American hostages in Tehran by Jan. 7. Four nations abstained.

The council acted shortly after Waldheim flew from New York to Europe en route to Tehran in a bid to persuade Ayatollah Ruhollah Khomeini's revolutionary government to resolve its seven-week confrontation with the United States over the fate of the hostages.

Voting for the resolution were Britain, France, China, Norway, Portugal, Bolivia, Jamaica, Nigeria, Zambia, Gabon and the United States. Abstaining were the Soviet Union, Czechoslovakia and two Islamic countries, Kuwait and Bangladesh.

[In Tehran, students holding the American hostages said the Security Council vote would not make them soften their stand and said that Waldheim would not be allowed to visit the hostages unless Khomeini ordered them to do so.]

[The students said they would not discuss the hostages with Waldheim, but left open the possiblity they would meet with the U.N. secretary general. "We are not afraid of economic sanctions," a student spokesman said. "They are not important for us or our people. We can stand it."]

Secretary of State Cyrus Vance, who led the U.S. delegation through three days of council debate and behind-the-scenes negotiations, hailed the outcome by saying:

"We are very, very pleased at the vote. We join in wishing the secretary general good fortune. We all hope he will have a fruitful trip."

In Washington, State Department spokesman Hodding Carter said: "Obviously, we hope the secretary general's mission will meet with success." He added, however: "The word 'hope' in this case has to be defined in the way you could say: 'I hope the year 1980 is going to be a wonderful year for all of us.'"

The general feeling at the United Nations after the vote seemed to be one of extreme pessimism about Waldheim's chances of breaking the impasse in the short time allotted to him under the resolution's mandate. Waldheim is scheduled to arrive in Tehran at noon local time Tuesday.

Iran so far has ignored a Dec. 4 Security Council resolution that was adopted unanimously and that calls for immediate release of the hostages.

As a result, U.N. sources predicted that on Jan. 7, the 15-nation Security Council will find itself unable to delay any longer the controversial task of trying to spell out the specific trade, financial and communications pressures that the United Nation's 152 members will be called upon to apply against Iran.

A complicating factor at that time will be the effects of the change Tuesday of five of the council's 10 nonpermanent seats under the U.N. system of rotating representation. The change poses major unanswered questions about how the new members will look upon sanctions and whether they will consider themselves bound by the resolution adopted today.

Although U.S. officials contend that today's vote commits the council to imposing sanctions if Iran remains defiant, the question of what specifically these measures will be also remains wide open.

Two of the countries voting for the resolution -- China and Zambia -- made clear that they are reserving their positions on sanctions. Zambia said flatly it does not consider itself bound by that part of the resolution, and China warned that it will approach any decision on sanctions with the view that the council "should take prudent attitude . . .. conducive to relaxation of tensions."

In addition, the Soviet Union, which abstained instead of using its veto to kill the resolution, hinted that it might still bring the veto into play if the council feels compelled to actually apply sanctions.

Soviet Ambassador Oleg Troyanovsky said the hostages' plight was "a bilateral problem between the United States and Iran." He charged it is "a distortion of the true state of affairs" to say the situation poses a threat to international peace under Chapter 7 of the U.N. Charter -- the basis for the council's action in the Iranian crisis.

"Attaching the question of sanctions is unjustified," Troyanovsky warned.

In response, U.S. Ambassador Donald McHenry said the issue "is not the question the Soviet Union has sought to put before us -- that there is some kind of U.S.-Iranian dispute. It remains a dispute between Iran and the international community."

Asked about the possiblity of a Soviet veto, McHenry cited the Soviet military involvement in the Afghanisan coup last week and noted that some U.N. members want to bring that issue before the Security Council. He added: "A country now engaged in the rape of another country would be ill-advised to engage in a veto."

But, when reporters asked about the possibility of a deal for the United States to soft-pedal the Afghanistan situation in exchange for Soviet cooperation on Iran, McHenry replied, "I consider that an obscene suggestion."

McHenry said the United States is in close consultation with both the continuing and new council members to define what actual sanctions might include. While conceding that certain "difficulties lie ahead," he expressed confidence that, if the need arises, the council will not shrink from putting appropriate sanctions into effect.

The resolution instructs Waldheim to report the results of his trip to the council by Jan. 7 and states that "in the event of noncompliance" by Iran, the council should "adopt effective measures" under those articles of the U.N. charter authorizing economic retaliation against countries whose actions are deemed a threat to international peace.

McHenry and other U.S. officials refused to specify what "noncompliance" means. But, the United States is expected to contend that Waldheim will have to secure by Jan. 7 either the actual freedom of the hostages or a firm promise by the Iranians of their imminent release.

Should specific sanctions be required, the United States is understood to be pressing for a boycott by U.N. members of all credit and all exports to Iran except food and medicines, refusal of port and landing facilities to Iranian ships and planes and a ban on shipment of military spare parts to Iran. The U.S.-proposed measures would exempt sales of Iranian oil to other countries.

In the past, most Iranian imports have come from the United States, Western Europe and Japan, and the Carter administration already has pledges from its principal industrial allies that they will use a U.N. sanctions mandate to cooperate with Washington in a concerted trade and credit squeeze on Iran.