The Environmental Protection Agency announced yesterday that it intends to restrict the use of tall smokestacks to comply with air-pollution standards -- potentially the most expensive environmental regulation to come out of the agency under the Reagan administration.

The action, forced by a court decision, could cost coal-fired power plants as much as $4.6 billion up front and increase their operating costs by as much as $1.4 billion a year.

It also would cut sulfur-dioxide emissions by as much as 2.9 million tons a year, nearly as much as EPA Administrator William D. Ruckelshaus had suggested last year as part of a pilot program to curb acid rain. That program was vetoed by the White House as being too expensive to warrant its uncertain results.

But EPA officials said yesterday's proposal was not aimed at controlling acid rain, and Assistant EPA Administrator Joseph A. Cannon told reporters that "if you were going to get that much sulfur out of the atmosphere, this isn't the cheapest way to do it."

The proposal is the latest chapter in a decade-long controversy over industry's use of towering smokestacks -- some as high as 1,200 feet -- to disperse pollution high into the atmosphere, rather than cut emissions or install costly pollution-control equipment.

Environmental groups have long contended that the tall smokestacks amounted to a circumvention of the Clean Air Act.

In 1982, however, the Reagan administration attempted to further relax the smokestack standards through a system of air-pollution "credits" for stacks that were tall enough to push pollution over downwind hills and mountains.

Those rules were struck down in October 1983 by the U.S. Court of Appeals here, which agreed with environmentalists that they did not meet the clean-air law's general ban on "dispersal" as a method of pollution control.

The U.S. Supreme Court denied an industry appeal of the decision last July.

Industry representatives immediately attacked the EPA's latest proposal, contending that their coal-fired plants are in compliance with clean-air standards and that the new rules will mean higher electric bills for their customers.

"This retroactive change would sharply raise costs with no identifiable benefits," the Edison Electric Institute said in a statement.

But environmentalists were skeptical of the proposal as well. "This is less than required by the law," said David Hawkins of the Natural Resources Defense Council, one of the groups that took the agency to court for its 1982 rule. "Our view is that they've approached the problem by giving as much credit as possible and are still trying to argue with a straight face that they've complied with the court order."

While the EPA's new proposal does not require industries to tear down their stacks, it sharply limits the amount of air-pollution credit given for stacks higher than 200 feet.

In practical terms, that means that many coal-fired utilities, smelters and other high-pollution industries will have to switch to low-sulfur coal, install stack "scrubbers" to remove pollutants or take other measures to meet air-quality standards.

According to the EPA, the new rules will affect more than 600 industrial plants, most of them in Midwestern states where high-sulfur coal is plentiful and relatively cheap.

An analysis conducted for the agency by ICF Inc., a Washington consulting firm, said the proposal could reduce the market for high-sulfur coal by as much as 20 percent in Indiana and 17 percent in Ohio and could increase electricity rates 2 to 4 percent.

Asked if the Office of Management and Budget had any reaction to the proposal, the EPA's Cannon deadpanned: "They were wildly enthusiastic about this. It's a very expensive regulation."