The European allies agreed tonight to impose phased economic and diplomatic sanctions against Iran between now and May 17 unless "decisive progress" is made toward the release of the American hostages in Tehran.

The sanctions will come in two stages, with an immediate ban on new contracts for exports or commercial services to Iran and on any remaining trade in weapons or military equipment. The nine nations of the European Economic Community will also immediately reduce the staffs of their embassies in Tehran, of Iranian embassies in European capitals and will require Iranians to obtain visas to travel in Common Market countries.

The final stage, to be imposed May 17, is a total embargo on all imports to Iran except food and medicine.

The program adopted is a slightly altered version of a British proposal, which had included the formal cutoff of all imports of Iranian oil and the possible recall of Common Market ambassadors in Tehran.

During difficult negotiations today, sources said, compromise proposals were adopted that fell short of immediate action on these two points.

British Foreign Secretary Lord Carrington said that the Europeans and Japan are already in effect doing "what the United States has asked" on the oil embargo question by refusing to pay more than average OPEC prices for Iranian oil. He said that British Petroleum, Shell and Japanese companies are no longer being supplied with Iranian oil because they refuse to agree to pay several dollars a barrel more than OPEC prices.

In Washington, State Department spokesman Tom Reston said he would "rather not get into what we've asked" the allies to do about Iranian oil imports, and said that while the United States "welcomed" Japan's decision last week not to pay the higher Iranian prices, that decision has been made by Japan on "purely economic grounds."

While issuing no formal statement, White House officials said last night that they were "pleased" with the outcome of the Luxembourg meeting.

The Common Market nations implied that the refusal to pay the higher prices was tantamount to a boycott. A West Germany source said that for his country, Europe's largest customer for Iranian oil, this means its imports will continue their already steady decline. By observing OPEC price levels, he added, "we are also telling OPEC that we are on their side."

Instead of breaking diplomatic relations with Iran, the allies are sending their ambassadors back to Tehran "to communicate the decision to the Iranian government, monitor developments and make every possible effort to improve the living conditions of the hostages until they are freed."

If the Iranians fail to make "decisive progress toward the release of the hostages" by May 17, the foreign ministers of the nine Common Market nations, who will be meeting in Naples May 17 and 18, are committed to the immediate imposition of economic sanctions contained in an U.N resolution approved last January by 10 members of the Security Council but vetoed by the Soviet Union.

According to Carrington, the ministers decided to delay triggering the tougher second stage of sanctions because several countries, including Britain, had to adopt legislation legalizing the imposition of those sanctions without a Security Council order. Carrington said this process would be completed by May 17.

"If there is no decisive progress, the sanctions will come into effect automatically," Carrington told reporters tonight. He and other British officials said they would not necessarily regard as decisive progress the transfer of the hostages to the Iranian government.

West German Foreign Minister Hans-Dietrich Genscher told the others that he expected it would be up to the United States to decide whether decisive progress has been made toward the hostages' release, according to a West Germany government source.

"There will be close consultations with the United States going on," he said. "They will tell us if it is decisive."

West German officials are satisfied that today's action irrevocably commits the European Community to speedy imposition of sanctions, this source said. West Germany, however, would still have preferred imposing complete economic sanctions today through Common Market trade powers, he said, rather than waiting for some of the countries to pass enabling legislation.

Sources said that eight nations -- Britain, West Germany, the Netherlands, Belgium, Luxembourg, Denmark, Ireland and Italy -- were, with varying degrees of enthusiasm, ready to act under Common Market authority. But France, according to these sources, held out.

The French foreign minister, Jean Francois-Poncet, insisted that it was unacceptable to use Common Market powers to regulate trade for political purposes such as imposing sanctions, which is consistent with the French position in the past.

Italian Foreign Minister Emilio Colombo also raised doubts about cutting off commercial relationships with Iran in addition to banning trade in the future. This debate held up proceedings for several hours tonight, and was described by many diplomats as primarily a technical and legal matter.

"But there was no question among all nine," a West German source said, "of the necessity of acting today" to commit themselves to impose sanctions jointly under each nation's own legal authority.

They reached that joint position only after becoming convinced during the past few weeks by the Carter administration's admonition that a strong European response on Iran could be vital to the future of the Western alliance and to avoidance of a possible military confrontation in the Persian Gulf.

European officials were alarmed, according to diplomatic sources, by the U.S. threats of military measures against Iran by mid-May if the allies did not help in increasing pressure on Iran.

Answering questions about the threat of military action, Lord Carrington said, "We understand the United States to say that if we all stuck together and we all took these measures, which we are now taking, then military action should not be necessary."