The Supreme Court technically came out on the government's side yesterday in its cottondust ruling, but in the process it also may have invalidated one of the main devices by which the Reagan administration had hoped to lighten the federal regulatory burden on business.

In rejecting the application of cost-benefit analysis to cotton-dust standards, the court warned the administration that it may not always be able to achieve its deregulation goals in the job-health field by just rewriting the rules.

Instead of this shortcut, in some cases it is going to have to take the long way through Congress, a path far less certain to take the administration where it wants to go.

The administration's "shortcut" is President Reagan's executive order of February, calling for a "regulatory impact analysis" describing the potential costs and benefits of every major regulation, defined as those likely to cost $100 million or more.

High on the administration's list of targets for this cost-benefit analysis were federal standards limiting employe exposure to toxic substances, which industry had complained were too rigid and expensive.

In his "dunkirk memo" to Reagan last November, Office of Management and Budget Director David A. Stockman recommended modifications in the standards for several toxic substances in the workplace, among them asbestos, chromium and cadmium.

The administration singled out the cotton-dust standards when it announced in March that it intended to review, and possibly relax, all federal standards limiting exposure to toxic substances.

In making the announcement, Thorne Auchter, head of the Occupational Safety and Health Administration, called cost-benefit analysis "an important tool in producing efficient regulation. . . . That's why we plan to apply cost-benefit analysis to the cotton-dust standard, to see how regulatory costs and benefits can be weighed in an actual case."

But the court ruled yesterday that the intent of Congress was to protect workers in any "feasible" way, and that any effort to balance the benefits of protection against the costs would be inconsistent with the law as passed by Congress.

An official of the administration's White House-level regulatory task force acknowledged that the decision was a setback or OSHA's deregulatory efforts, but said the government-wide effort to impose cost-effectiveness or regulation will continue.

"It would be fair to say that we're in some ways disappointed in it," said task force counsel C. Boyden Gray. "But it doesn't affect the overall effort."

Gray said the ruling "seems to preserve the executive order and the things we want to do at OSHA." The executive order, he noted, does not require cost-benefit analysis where forbidden by statute, on emergency regulations or on regulations being issued to satisfy court orders.

"The executive order says we can only do what's allowed by law," he said.

As far as the cotton-dust issue goes, he said, "The court has spoken on that. Cost-benefit analysis cannot be used under that statute."

In a statement yesterday, OSHA's Auchter claimed victory for both OSHA and for the cost-benefit principle.

The decision in the case, which had been submitted to the court under the Carter administration, "achieves a major goal, that is, it maintains the cotton-dust standards in effect, protecting textile workers," he said.

But at the time, he said the court also supported the argument that "long-term profitability and competitiveness of the industry is an appropriate test in determining what is 'feasible.' . . . The court suggested that the agency may be required to use cost-effectivenss studies to determine what particular methods should be used to achieve the required levels of protection."

Auchter noted that the court " did not specifically exclude" cost-benefit analysis in areas other than health standards, and did not decide the legality of such analysis under other statutes.

The American Textile Manufacturers Institute, plaintiff in the cotton-dust suit, also attempted to put the best possible face on the ruling. A spokesman said the institute was "disappointed in the decision."

"We continue to believe, as does the adminitration, that cost-benefit analysis is an important element in achieving rational and cost-effective rule-making," said spokesman Jack Glawson.

"And the Supreme Court did not say otherwise. All that a majority of the court did say is that the Occupational Safety and Health Act does not require OSHA to engage in cost-benefit balancing when it sets health standards for toxic substances."

But AFL-CIO Secretary-Treasurer Thomas R. Donohue, whose union members in the textile industry will benefit from the court's ruling, said the ruling means "you can't invent a cost-benefit analysis and somehow inject it into the law." At a news conference yesterday, Donohue said the court "has hopefully put to bed the concept that the measure for the effect of a statute is cost-benefit analysis."

OSHA is still reviewing a number of standards for other toxic substances in the workplace, and administration officials yesterday were unwilling to speculate on whether those reviews will be halted or, if necessary, legislation introduced to permit cost-benefit analysis.

Meanwhile, lawyers were still debating the extent of the ruling -- if, for instance, the2 court intended to permit the use of cost-benefit analysis in areas other than those concerning health standards.

"You can be sure this is not the last case to be litigated under the OSHA act," said a lawyer for the textile industry.