The Supreme Court, rebuffing the Reagan administration in a decision with potential repercussions throughout the regulatory field, yesterday refused to east stringent occupational safety and health laws to save money for business.

Deciding one of the most debated regulatory questions in a decade, the cost-benefit controversy, the court ruled that regulators are not required to temper worker protections against hazardous and poisonous substances to make their costs reasonable to industry.

The health of workers outweight "all other considerations," Justice William J. Brennan wrote in the 5-to-3 decision upholding textile factory cotton dust exposure standards designed to combat brown lung disease.

Congress intended that compliance be "part of the cost of doing business," he said. "Any standard based on a balancing of costs and benefits . . . would be inconsistent" with the law, Brennan said.

The majority pointedly refused a Reagan administration request that the court withdraw from the controversy while the Occupational Safety and Healty Administration experiments with cost-benefit analysis for regulated industries.

The decision did not say specifically that cost-benefit analysis is prohibited. But Brennan said it would be contrary to the intent of the law, and lawyers debated yesterday whether that wording could be read as a prohibition by lower court judges in future cases.

Unless Congress answers it, that question undoubtedly will set off a new round of court cases, which itself could sidetrack Reagan's regulatory program, in which cost-benefit analysis is a centerpiece.

Thorne Auchter, Reagan's new OSHA head, tried to hail the ruling as a victory, apparently because the agency defended the regulations before the Supreme Court during the presidency of Jimmy Carter.

Reagan's OSHA, however, announced in March that it would apply cost benefit analysis to the cotton dust regulations and asked the justices to withdraw from the case. Brennan dismissed that request in a terse footnote yesterday.

The textile industry saw the decision largely as a defeat. "We're disappointed," said a former statement from the American Textile Manufacturers Institute, which challenged the regulations.

"We're on cloud nine," said Florence Sandlin, a retired textile worker active in the Brown Lung Association in Greensboro, N.C., and the wife of a brown-lung disease victim. "We're going to celebrate."

Joining Brennan were Justices Byron R. White, Thurgood Marshall, Harry A. Blackmun and John Paul Stevens. Justice William H. Rehnquist, Justice Potter S. Stewart and Chief Justice Warren E. Burger dissented. Justice Lewis F. Powell Jr., without explanation, did not participate.

Byssinosis, or brown-lung disease, is a chronic respiratory disease similar in its symptoms to emphysema. In its mild forms, it saps its victims of energy and often makes it impossible to work, walk for long distances or have sexual relation. In its most severe forms, it can cause heart ailments and kill. The industry estimates that fewer than 2 percent of its workers are victims. Non-industry studies put the figure at 25 percent.

The cause is cotton dust, the stem and leaf material that sticks to the cotton plant when it arrives at textile mills. As the plant is processed into yarn by high-speed machinery, the dust flies into the lungs of textile workers.

OSHA imposed requirements in 1978 to bring down permissible cotton dust levels from 1,000 micrograms per cubic meter of air to 200 micrograms for the most hazardous sections of the mills. That level is said to be the equivalent of a grain of salt in a box of air a meter (39.37 inches) long on each side.

Industry said that the equipment -- giant vacuum devices and elaborate ductwork -- required to meet that standard was too expensive and might even put smaller firms out of business. It also said there were cheaper ways of reducing the incidence of the disease, such as careful medical screening.

The textile manufacturers' institute and 12 major textile companies sued OSHA, saying the Occupational Safety and Health Act of 1970 required OSHA to redo its regulations to determine that the costs to industry bear a reasonable relationship to its benefits to workers.

It was the same theme struck by numerous other industries in other court challenges against OSHA regulations covering exposure to such substances as lead and benzene.

A panel of the U.S. Court of Appeals for the District of Columbia, in an opinion written by Judge David Bazelon, rejected the challenge to the cotton dust regulations and held that cost-benefit was not required.

Last term, the Supreme Court, ruling on OSHA's benzene regulations, met industry halfway be deciding that the law required a showing of "significant risk" to worker health in promulgating regulations. But it was unable to reach agreement on the cost-benefit issue.

Yesterday it did. Upholding all but one minor element of Bazelon's ruling, Brennan cited the section of the law governing regulations for toxic materials and harmful substances: OSHA, it says, "shall set the standard which most adequately assures, to the extent feasible . . . that no employee will suffer material impairment of health . . . ."

The dictionary, Brennan said, defines "feasible as capable of being done, executed or effected."

Congress "meant feasible and nothing else in using that term," Brennan said. "The legislative history demonstrates conclusively that Congress was fully aware that the act would impose real and substantial costs of compliance on industry and believed that such costs were part of the cost of doing business."

Congress "intended to impose such costs when necessary to create a safe and healthful working environment," he wrote. It did not "contemplate any further balancing by the agency. . . ."

The court did not offer a definition of "feasibility." OSHA had decided that industry could implement its proposed cotton-dust standards without a severe threat to its profitability. The study on which this decision was based was far less rigorous than the elaborate cost benefit analysis in question.

But Brennan said that OSHA's effort is "certainly consistent" with the law. That question is expected to prompt more litigation.

Brennan did overturn one part of Bazelon's ruling. Osha may not necessarily require that textile workers moved within a plant to avoid cotton dust be held harmless from any loss of benefits.

Rehnquist, joined by Burger, dissented fromthe majority ruling, stressing that it did not prohibit cost-benefit analysis. They repeated their theme, spelled out in earlier cases, that Congress should have answered these questions, not the courts.

Stewart said the case should have been settled without deciding the cost-benefit question. Rather, he said, he would overturn the regulations because OSHA had failed to justify its estimates of the cost to industry.

Textile industry lawyers found some comfort in the opinion. They said that cost-benefit analysis was not prohibited, though they conceded that would be a matter of debate. They noted that the court appeared to leave room for the use of cost-benefit analysis in determining which chemicals should be regulated, if not how to regulate them.

And OSHA officials underscored the court's statement that it was not ruling out a cost-benefit requirement for regulations of job safety unrelated to toxic and hazardous exposure.

In other action yesterday:

The court upheld the government's application of one of the most widely used federal laws aimed at business corruption. Novia Turkette and others charged with narcotics trafficking challenged the Racketeer Influence and Corrup Organizations (RICO) statute on the grounds that it was not meant for use against wholly illegitimate enterprises, such as narcotics rings.

They argued that the law was primarily designed to combat the infiltration of legitimate businesses by racketeers.

The U.S. First Circuit Court of Appeals agreed, and the Justice Department, saying that the ruling could damage its effort against racketeering, sought the reversal.

The court ruled 8 to 1 yesterday in U.S. vs. Turkette that the law applied to both legitimate and illegitimate enterprises. Stewart dissented.

The justices also ruled in a Connecticut case that inmates are not constitutionally entitled to a written explanation when a pardons board refuses to commute their sentence.

The issue was one of "entitlement" under the due-process clause of the Constitution. A lower court held in Connecticut Board of Pardons vs. Dumschat that, because the state's board of pardons often commuted life sentences, inmates had a protected "liberty interest" in the board's proceedings which mandated written explanations of denials.

Burger, in the 7-to-2 decision, disagreed. The fact that the pardon board often commuted sentences does not create a right to any sort of formal proceeding. "Statistical probabilities standing alone generate no constitutional protections."