Secretary of State Cyrus R. Vance charged today that some Japanese companies are rushing to buy, "at extraordinarily high prices," Iranian oil formerly sold to the United States. He also asserted, according to U.S. officials, that Japanese banks are helping Iran overcome the economic squeeze imposed by the freezing of Iranian assets in American banks.

Vance, seeking the help of major U.S. allies for joint economic pressures on Iran, warned Japan that the United States thinks these Japanese actions could harm efforts to free the 50 American hostages in Tehran.

U.S. officials said that at a meeting here tonight Vance told Foreign Minister Subaro Okita that the activities of those Japanese banks and oil companies are "sending ambiguous signals" that might encourage Iranian leaders to ignore international calls for release of the captives.

Vance's reportedly blunt message to Okita came in the course of a fast-paced day that saw the secretary stop in London for talks with British Prime Minister Margaret Thatcher and then travel here for a dinner meeting with French President Valery Giscard d'Estaing. On Tuesday, Vance goes to Rome and Bonn for similar talks with Italian and West German leaders.

He is making the trip to explore the possibilities of coordinated international economic retaliation if Iran continues to defy the United Nations Security Council call for freeing the hostages.

In The Hague, meanwhile, the United States brought its case against Iran before the International Court of Justice today, calling for a swift end to Iran's "barbaric" detention of the hostages.

[Attorney General Benjamin Civiletti told the 15 justices in a two-hour hearing boycotted by Iran that they should order the release of the hostages, "save their lives" and restore the rule of internatioal law. Story, Page A11.]

As part of the U.S. campaign to tighten the screws on Iran's revolutionary government, the Carter administration is trying to prepare the way for an informal trade embargo and other sanctions by the major Western industrial nations.

In the early stages of the 37-day-old crisis, the United States employed its own economic leverage by suspending indefinitely all imports of Iranian oil and freezing the approximately $9 billion in Iranian assets held by U.S. banks and their overseas subsidiaries.

Now, the administration, assuming that increased economic disruptions will increase the domestic problems confronting Ayatollah Ruhollah Khomeini and make Tehran more flexible in negotiating about the hostages, wants its major allies to help demonstrate that Iran cannot circumvent the U.S. pressures by turning to other industrial countries.

Senior U.S. officials traveling with Vance said he is trying to accomplish two specific goals on his mission: first, to ensure that those measures instituted by the United States continue to be effective, and second, to discuss with Washington's allies what steps they might be willing to take if the Iranians remain intractable.

According to the officials, it was the first consideration that led Vance to put Okita on notice about Washington's displeasure with moves by Japanese business to undermine the U.S. sanctions. d

In recent days, the officials said the United States had found "strong evidence" that Japanese firms have shown what one source called "unseemly haste" in rushing to buy the oil rejected by the Untied States on the spot market at prices of almost $40 a barrel. That, the officials said, was being done despite the fact that Japan has stockpiled unusually large amounts of oil and despite Japanese promises at the Tokyo economic summit last summer to join efforts to reduce oil imports.

In addition, the officials continued, Washington has evidence that Japanese banks are giving Iran technical advice and facilities designed to help it obtain and pay for essential imports it formerly received from the Untied States.

The officials said Vance expressed understanding for Japan's situation as a country that must import almost all of its energy requirements. As a result of its recent heavy spot market purchases, Japan gets almost 20 percent of its oil imports -- roughly 14 percent of its total energy needs -- from Iran.

Nevertheless, the officials added, Vance also stressed that the United States and other industrial countries associated with Japan in the International Energy Agency are pledged to help Japan overcome any shortfalls it might encounter as the result of emergencies such as the current turmoil in Iran. Okita is in Paris for the meeting of the IEA.

In response, the officials said, Okita noted that Japanese government is limited, in legal and practical terms, in the steps it can take against the companies and banks whose practices were cited by Vance. However, the officials added, he did promise that his government would attempt to intercede with the Japanese business community to try to make it more cooperative with U.S. economic campaign against Iran.

As to possible other steps that the industrial countries might be willing to take to support the U.S. position, the officials were more guarded.

Although they described the meetings with British leaders in London this morning and with French leaders here tonight as "very cordial," the officials refused to say whether any agreements had been reached. Instead, they stressed, Vance's talks were concerned mainly with things that might happen in the future, and they cautioned against expecting any "big headlines or sudden pronouncements" from the capitals he is visiting.

All of America's major allies have joined publicly in calls for release of the hostages, and some have tried to intercede with the Iranians through private channels. But, as countries dependent on oil imports and with major business interest in the Middle East, there are big questions about how far they would be willing to go in applying economic sanctions against Iran.

Britain, West Germany and Italy have cut off the supply of military spare parts to Iran -- a move welcomed by the Carter administration. But, most of the industrial nations have made clear that they would not want to see major disruptions in world trade if they can be avoided.

However, these countries also are extremely nervous about the United States possibly having to contemplate military action against Iran. Carter has said he does not intend to use force if it would endanger the hostages, but all European governments are keenly aware that situation could change if the hostages are put on trial or if the situation drags on to the point where the American public becomes overly restive.

U.S. officials stressed that Vance is not using that argument in his talks here. However, the unspoken possiblity that the Iranian crisis could erupt into bloodshed and a military response is probably the strongest card the United States has in its bid to draw its allies into concerted action on economic sanctions.