Major Western European countries and Japan appeared to be moving today toward imposition of tough economic and diplomatic sanctions against Iran, beginning with a boycott of Iranian oil and severe cutbacks in diplomatic representation in Iran.

If these moves failed to produce progress toward release of the American hostages within a month or so, the second stage of sanctions, including a near-total trade cutoff except for food and medicine, would be imposed, according to the British-sponsored plan.

Well-informed British sources said tonight that they are "confident" that this plan, with some modification, can win approval tomorrow from the foreign minister of the European Common Market meeting here.

They said Britain also was seeking the cooperation of Japan, whose foreign minister met informally here today with European officials, including British Foreign Secretary Lord Carrington, to discuss ways of assisting the U.S. moves to bring pressure on Iran.

[The Australian government clamped down on trade with Iran Monday, making it the second Western nation to join the United States in imposing economic sanctions against Iran. Portugal halted trade with Iran last week.]

[Meanwhile, as Iran moved to cut off its oil exports to Japan, officials in Tokyo said they would seek other sources of oil and may ask the United States for aid.]

The first stage of the British proposal calls for the Common Market countries and Japan to reduce immediately their embassy staffs in Iran and perhaps recall their ambassadors, reduce the number of Iranian diplomats in European capitals, formalize a ban that most Common Market countries have already imposed on military sales to Iran and begin a coordinated boycott of Iranian oil.

Britain, which extracts oil from the North Sea, also "would be willing to engage in consultations about ways in which the shortfall might be made up" for Europe and Japan, a British source said tonight.

The Carter administration has made a similar offer.

Together, the nine Common Market countries import only about 5 percent of their oil from Iran, although West Germany and Ireland import about 10 percent.

If the first stage actions failed to improve the hostage situation, the British plan then calls for imposition of the rest of the economic sanctions sought by President Carter -- but not a complete break in diplomatic relations with Iran.

British sources said two stages were required to allow Common Market countries to adopt the necessary legislation to impose such sanctions. They denied that the British plan was intended as a compromise between the tougher West German and more conciliatory French positions, although other diplomats here nevertheless see it as just that.

Britain's proposal has been discussed in detail with the other Common Market countries in recent days British sources said, and "was well received in the other capitals."

Diplomats gathered here for the European foreign ministers' meeting indicated that Ireland, Denmark, Belgium, the Netherlands and Luxembourg are ready to support the British plan. West Germany's only reservation, according to Bonn diplomats, is its desire to impose complete economic sanctions sooner.

France and Italy reportedly would like to delay economic sanctions longer, after making one more diplomatic plea to Iranian President Abol Hassan Bani-Sadr. But a ranking French diplomat indicated here tonight that the French were ready to accept the British proposal with some modification.

Japanese Foreign Minister Saburo Okita told reporters here today the Japanese position "is very close to that of the Europeans. It is very important to maintain solidarity between Europe, Japan and the United States without resorting to military measures" against Iran.

The British acknowledge European objections that sanctions may well fail to help free the American hostages and could drive Iran into the arms of the Soviet Union. The European press has emphasized recent trade negotiations between Iran and the Soviet Union and Warsaw Pact countries.

But British sources close to Carrington said tonight that "something has to be done even if it does not work. It is our strong hope that the psychological impact of this plan plus what the U.S. is doing will work."

They also acknowledged the concern of Carrington, along with other European diplomats, that Carter would resort to military action if the Europeans did not act soon to give him sufficient support. But they insisted that Europe was not under an ultimatum or had "a pistol put to our head."

At the same time, European diplomats have repeatedly referred to the dangers they saw in potential U.S. military action in the Persian Gulf and the great strain that disagreement over the efficacy of sanctions against Iran has put on the Atlantic alliance.

"This is almost a spiritual crisis in the alliance, an emotional crisis," Britain's deputy foreign minister, Douglas Hurd, said in London yesterday.

"It's not just a matter of pleasing President Carter or getting ourselves out of the line of fire of American newspapers. The United States needs to be assured there are people in Europe who understand what it is going through."

"The price of delay would just be too high for the alliance," a West German source said here today. "We must not divide ourselves from the Americans, but we do not want to divide the European community either. That is why we want to act soon but within the community if it is at all possible."

The Associated Press reported from Canberra:

The Australian restrictions announced today do not affect food exports -- the most important sector of Australian sales to Iran -- and fall short of a total embargo on other goods. But they will make Australian export deals with Iran difficult or unprofitable.

Prime Minister Malcolm Fraser's government will deny export incentives, subsidies and tax concessions to any firms exporting nonfood items to Iran, and will limit insurance coverage on such shipments.

Australian nonfood exports to Iran in the first seven months of the 1979-80 fiscal year were worth about $12 million, compared with $105 million in food exports.