The Supreme Court headed off expected efforts to get thousands of private patent monopolies in the vast computer program or "software" industry by ruling yesterday that an inventor can't patent a method of identifying a limited category of useful but conventional applications when the only novel feature is a mathematical formula.

The case involved the warning alarms used in petroleum refining or other systems for the catalytic conversion by hydrocarbons in which changes occur in the permissible limits of heat, pressure, and flow rate as the process proceeds.

In 1971, Dale R. Flook, an employe of Atlantic Richfield Co., applied to the U.S. Patent and Trademark Office for a pattent on a method to use data gathered from sensors to modify the "alarm limits" - which are numbers - as the process proceeds.

The method entailed programming a computer with an algorithm, which is a procedure for solving a given type of mathematical problem. The novelty in the application lay entirely in the algorithm. All of the other components - the sensors, the computer itself, the devices for adjusting the alarms - were commonly known.

The stakes were high, partly because incentives of patents were available for computer hardware but not for software, and some software producers complained that this retarded their progress.

In 1974, the software industry employed 195,000 computer programmers. In 1976, the cumulative value of computer programs was estimated at $43.1 billion and gross revenues from sales of software were $1.6 billion. Estimates are that the cumulative value will reach $70.7 billion in 1980 and that sales will reach $5.6 billion by 1981.

A patent examiner rejected the application on the ground that Flook couldn't patent a formula for computerizing computations that could be done with a pencil and paper. An appeals board in the agency affirmed but was reversed by the U.S. Court of Customs and Patent Appeals.

Yesterday, the Supreme Court overruled the lower court 6 to 3 with an opinion by Justice John Paul Stevens.

The result was the one sought by the government, which said that the lower court's decision would have a debilitating effect on the software industry.

Stevens wrote that the line in the patent code between a patentable "process" and an unpatentable "principle" doesn't always "shimmer with clarity."

But saying that the government could withhold a patent from Flook, Stevens cautioned against any assumption that patent protection of certain novel and useful computer programs will not promote the progress of science and the arts, or that such protection is undesirable as a matter of public policy." Congress can change the law if it cares to, he said.

In the dissenting opinion, Justice Potter Stewart, joined by Chief Justice Warren Burger and Justice William Rehnquist, charged that the court struck a "damaging blow at basic principles of patent law by importing into its inquiry . . . the criteria of novelty and inventiveness," when the law "is concerned only with subject matter patentability."