The Security Council president, with unanimous backing from the other 14 members, told Iran today that the body would not meet to hear Iran's complaint against the United States until the American hostages were freed.

Just such a decision had been sought by Secretary of State Cyrus Vance who met today with the council president and U.N. Secretary General Kurt Waldheim.

The hope of the United States, and other council members, is that Iran will enter a negotiating process through the United Nations -- but a process that will center on the release of the hostages, and only afterward provide Iran with a forum to state its case.

Asked whether he considered a council meeting inappropriate when the Americans remained in captivity, Vance told reporters here:

"We believe the hostage question is the question which is before all of us. We have indicated that if the hostage question is resolved, then we are propared to see the Security Council go forward and deal with the issues anyone wishes to put before us, including the Iranians."

Vance spoke outside the U.S. mission, where he conferred with the American delegation to the United Nations. Later, he spent an hour at Waldheim's Sutton Place residence with the secretary general, Security Council President Sergio Palacios de Vizzios of Bolivia and General Assembly President Salim A. Salim of Tanzania.

There was no formal rebuff of Iran's request for a council meeting to discuss a "war psychology" in the United States, because most council members the Americans included, wish to preserve the principle that any country can get a hearing before the council if it wants one.

In theory, it takes nine of the 15 members to approve the convening of the council, but in this case all 15 were agreed that the best tactic was to press the Iranians for a response on the hostages as a delaying tactic.

Both the Western countries and the nonaligned nations took the position in private consultations that it would be inappropriate to hear Iran's complaint until Tehran had responded to Friday's appeal by the council for the release of the hostages.

The ranking Iranian diplomat here, Jamal Shemirani, said in an interview that there has been no reply to the council appeal because "it was not a reflection of reality. We did not have an opportunity to express our views before the council in advance of the statement by the president."

Shemirani asked for an appointment with Waldheim tonight, on instructions from Tehran, to seek a formal response to his government's request for a council meeting.

Early in the day, U.N. officials said they had been told informally that Iran's acting foreign minister, Abol Hassan Bani-Sadr, would come for the council debate. The U.N. officials said later that it was no longer clear that Bani-Sadr would be coming until the prospect of a council meeting was clarified.

Since clarification was the very thing the United States and other council members wished to avoid, the prospect of this particular negotiating tactic remained in doubt.

One Western member of the council suggested another reason for the tactics.

"We have our own prestige to consider," he said, "and we deserve a reply from Iran."