The Labor Department has beaten a surprise retreat from its proposed firing of a specialist in cancer-causing substances, thereby cooling a congressional inquiry, easing concerns over freedom of scientific expression in the Reagan administration and heading off a prolonged legal battle.

In a conciliatory letter Friday to the Occupational Safety and Health Administration scientist, Dr. Peter F. Infante, OSHA Administrator Thorne G. Auchter dismissed tentative charges of insubordination and of misrepresentation of the agency's position in an episode involving the carcinogenic potential of formaldehyde, an industrial chemical.

The letter came three weeks after Auchter and Infante's boss contradicted each other's sworn testimony in a House Science subcommittee hearing.

Infante, a GS15 who has headed OSHA's Office of Carcinogen Identification and Classification for 31/2 years, said he was "delighted" to be able to return to "my job of protecting the public -- and particularly workers." Dr. Sidney M. Wolfe, head of Public Citizen Inc., which represented Infante, hailed "an important victory for scientists and workers over the Reagan administration's efforts to suppress free scientific speech."

Subcommittee Chairman Albert Gore (D-Tenn.) declared "a victory for the integrity of science and for the right of free expression," and commended Auchter's probably "difficult" but "right" decision. The subcommittee has been trying to "clarify" the reasons for the contradictory sworn testimony given by Auchter and Infante's supervisor, Dr. Bailus Walker, who recently left to become Michigan's public health chief.

Walker said that he had wanted only to caution Infante informally, but that Auchter insisted on a formal letter to Infante proposing his dismissal. Auchter denied that he had ordered the firing.

Infante had been targeted by the Formaldehyde Institute, a trade group that wrote Auchter aide Mark Cowan to complain. Institute lawyer S. John Byington challenged unpublished New York University Medical Center research data showing the chemical to be a carcinogen in animals, citing a letter signed by Harry B. Demopoulos, associate professor of pathology at the center. Since then, however, Dr. Arthur C. Upton, former head of the National Cancer Institute who now heads the N.Y.U. department where the research was done, has accused Demopoulos -- in a letter to the Consumer Product Safety Commission -- of "misrepresentations," "inaccuracies" and statements that are "fallacious and simply not true."

Walker agrees with Infante that formaldehyde has been shown to be a carcinogen in animals and a cancer risk in humans -- the view also of the director of the National Cancer Institute, three other top federal toxicology experts, and the National Institute of Occupational Safety and Health (NIOSH).

The attempted firing began May 12, when Infante protested to the International Agency for Research on Cancer the conclusion of an IARC task force that published data were inadequate for an evaluation of formaldehyde's possible cancer risk in humans. His use of an OSHA letterhead became a reason for the attempted firing, but Auchter now says this was "at most a minor infraction."

To support his protest, Infante enclosed a publication of NIOSH, which is not a Labor Department agency. IARC director John Higginson charged, and Infante denied, that Infante was trying to speak for OSHA and had acted improperly.

Then came the attempted dismissal letter. Its key charge: Infante had insubordinately misrepresented Auchter's decision to withdraw OSHA as a cosponsor of the NIOSH publication. Auchter now admits that "documentation does not exist" for the charge that Infante had been told of the decisions.