A federal judge cleared the way yesterday for the Reagan administration to penalize a U.S. firm for violating President Reagan's embargo against the proposed Soviet-Western Europe natural gas pipeline, even as senior administration officials were planning to impose such penalties.

U.S. District Court Judge Thomas A. Flannery rejected a temporary restraining order requested by Dresser Industries of Dallas and its subsidiary, Dresser France, saying the firms had failed to prove they would be "irreparably damaged" by embargo sanctions or that they had a reasonable chance of winning their case in a trial.

Dresser France is moving to supply three giant pipeline compressors to the Soviet Union under orders from the French government. But Dresser's home office in Dallas has been ordered by Washington not to supply the equipment.

Flannery said that the case "involves a very serious area of foreign affairs" and that the firms had failed to make "an extraordinarily strong showing" that damages would be sufficient to justify intervention.

Demonstrating the gravity of the case from the administration viewpoint was a 1 1/2-hour meeting at the State Department, chaired by Secretary of State George P. Shultz and attended by three Cabinet secretaries and other top government officials.

No decisions were announced after the session, and officials were unusually secretive about everything connected with it, including the names of those who attended.

Administration sources said no announcements are likely for a day or two, in the hope that diplomatic means might be found to head off a damaging legal and political confrontation between the United States and its European allies. The administration is expected to be in contact shortly with the Europeans through diplomatic channels or a special U.S. mission.

"I don't think we have yet reached the stage where the situation is locked into a Greek tragedy," an official said.

The sources conceded, however, that the odds are against forestalling shipment of three Dresser-made compressors waiting at the French port of Le Havre to be loaded on a Soviet freighter.

If the shipment goes ahead, the sources said, the administration is preparing to impose "measured civil penalties" against Dresser for failing to comply with the U.S. Export Administration Act.

Government lawyers are said to be working on details of such action, which could involve disqualifying Dresser and its French subsidiary from receiving U.S. export licenses necessary in their international business.

A final decision on the U.S. penalties will have to be taken by President Reagan, who is vacationing in California. Reagan was briefed on the results of the Washington meeting, but White House spokesman Larry Speakes refused to say what the president was prepared to do.

Among those who attended the State Department meeting, in addition to Shultz, were Defense Secretary Caspar W. Weinberger, Commerce Secretary Malcolm Baldrige, Agriculture Secretary John R. Block, special trade representative William E. Brock, Deputy Treasury Secretary R.T. McNamar and Murray L. Weidenbaum, retiring chairman of the Council of Economic Advisers.

Baldrige and other Commerce Department officials had been the target of the legal action in which Dresser asked the federal court to block any U.S. government retaliation for delivery of the pipeline compressors. Dresser attorneys indicated after the ruling that they are considering an appeal to a higher court.

The French government, in the first formal challenge to Reagan's anti-pipeline sanctions June 18, ordered Dresser France Monday to deliver the compressors, valued at $2 million.

John Vanderstar, a Dresser attorney, said the firm, caught between the demands of two governments, is in "a terrible jam."

"Dresser France is being told by the French government to honor its contract, or else," he said. "Dresser France is being told by the U.S. government not to honor its contract, or else."

Arguing the government's case, Richard Willard, acting head of the Justice Department's civil division, said Dresser is asking the judge to say "that France is right and the president is wrong" in the case.

Penalties open to the administration include fines, criminal and civil actions, and placing the companies on a "blacklist" that could prohibit the American firm from receiving export licenses and its French subsidiary from receiving goods and data from the United States.

Dresser France contracted with two French firms last September to manufacture 21 compressors for use in the Soviet-West European natural gas pipeline for about $14 million. Using French workers and technology developed by its U.S. parent firm, Dresser France has built three compressors and has 18 in various stages of construction.

Last December, three months after the French subsidiary contracted for the compressors, the administration banned the export to the Soviet Union of oil and gas transmission equipment manufactured by U.S. firms or containing parts manufactured in this country. The action was taken in retaliation for Soviet involvement in the imposition of martial law in Poland.

Reagan, in a decision he linked to the absence of improvement in the martial-law situation, expanded the embargo in June to include foreign firms if they are owned or controlled by American firms or if they use any technical data developed by a U.S. firm.

Dresser officials said the French affiliate and its managers would be subject to criminal penalties, including a $60,000 fine and up to 12 months in jail, for failing to comply with the order from Paris.

On the other hand, U.S. penalties for failing to comply with Washington's order could put a serious crimp in the parent firm's $1.5 billion annual sales in international trade, a little more than one-third of the giant firm's total business.

John Brown Engineering Ltd., a British firm, is under contract to produce 21 turbines for the Soviet-European pipeline using General Electric rotors and some other GE-licensed elements.

The firm is expected to begin loading the first of the turbines, under British government instructions, later this week, spreading the legal confrontation to a second U.S. ally in Europe.