The U.N. Security Council moved toward agreement tonight on a resolution calling for Iran to release its 50 American hostages and use the United Nations as a forum for peacefully resolving its conflict with the United States.

The 15-nation council tonight ended its third day of emergency debate on the Iranian crisis without a resolution actually being tabled. Diplomatic sources said, however, that the necessary agreements in principle have been made in intensive backstage negotiations and that introduction and unanimous adoption of the resolution is expected when the council reconvenes at 4 p.m. Tuesday.

Although some points were still being negotiated, the sources said the resolution's main thrust will have a dual purpose:

To emphasize to Iran that its action in using diplomats as hostages has no support in the world community and should be abandoned immediately in favor of a return to accepted standards of international law.

To entice Iran, which has been boycotting the council meeting, into using the United Nations as a channel for seeking peaceful resolution of its grievances against the United States and deposed shah Mohammad Reza Pahlavi.

The crisis began Nov. 4, when Iranian militants, backed by the ruling authorities seized the U.S. Embassy in Tehran and took the Americans there hostage to enforce their demand that the United States forcibly return the shah, who has been undergoing medical treatment in this country, to stand trial as a criminal.

The sources said the resolution will contain three main points: A call for release of the hostages, instruction to Secretary General Kurt Waldheim to continue using his "good offices" in an effort to resolve the dispute and an appeal to the United States and Iran to settle their difference peacefully in accordance with the principles of the U.N. Charter.

Whether the resolution will make its intended impact on Iran's revolutionary leaders is still far from clear, the sources conceded. But, they added, the intention is to put the stress on conciliation rather than condemnation, and the hope here is that the Iranian authorities will see it as an opportunity to get off their current collision course with the United States without losing face.

As one source here put it, "The wording of the resolution is intended to take the general shape of an olive branch rather than a clenched fist."

Regarded in this light, such a resolution, if adopted by the council, would mark a success for the U.S. policy objectives in the debate. The Carter administration has been seeking to keep world attention focused on the plight of the hostages, while emphasizing its willingness to negotiate outstanding diffrences with Iran once the captives have been safely freed.

In the view of the Carter administration, the unfolding of the debate, which included unanimous criticism of Iran's actions by countries from every part of the political and geographical spectrum, has, at the least, demonstrated U.S. willingness to use every available forum to resolve the crisis peacefully.

Although they are less certain that the results will go beyond putting that point into the record, U.S. officials also clearly hope that Iran will heed the signals contained in the resolution and begin edging toward some form of U.N. mediation.

Specifically, the best hope in the view of most diplomats here is for the possible revival of Waldheim's efforts to work out a three-part deal -- release of the hostages in exchange for Iran's receiving the opportunity to state its grievances before the world body and appointment of a commission to investigate the shah's alleged crimes.

Although these elements are hinted at in the proposed resolution, the United States, in line with its stance of making no deals until the hostages are freed, has been fighting behind the scenes to keep the resolution as nonspecific as possible on actual promises to Iran. For that reason, the resolution is not expected to refer specifically to such possible avenues of redress for Iran as appointment of an international commission.

In addition, the United States also was known to be fighting tonight to keep out of the resolution language that would make specific reference to not using force and respecting the territorial integrity of other countries.

Such language reportedly is being backed by some Third World countries and the Soviet Union. U.S. officials here are understood to be arguing for more general, less binding language such as a reference to "the principles of the U.N. charter."

In the resolution is adopted as expected, Waldheim then would be the potential principal conduit for trying to arrange some kind of dialogue between Washington and Tehran. On Sunday, the secretary general spoke of his belief that Iranian Foreign Minister Sadegh Ghotbzadeh wants a negotiated settlement and hinted that Ghotbzadeh was sending a special emissary to New York to explore the matter further.

Later, after Ghotbzadeh denied that the man being send here would be empowered to negotiate for Iran on the hostage seizure, Waldheim backed away and said that he would have to see "what kind of mandate" the Iranian has.

As a result, there was confusion here about whether Iran is moving toward behind-the-scenes negotiations or whether it intends to continue its defiant no-compromise boycott of the U.N. effort to act as a mediator.

Sources in the U.S. delegation said tonight that they have seen no real signs either in Waldheim's contacts with Ghotbzadeh or though other channels that the Iranians are willing to start negotiating. But they added it remains to be seen how the Iranians respond after a resolution is adopted.