The Reagan administration is preparing to relax the federal regulations that limit worker exposure to cotton dust, thereby giving the textile industry by administrative order what it has been seeking unsuccessfully in the courts for years.

Government sources said the Occupational Safety and Health Administration will announce today its intention to amend the regulations to require that the cost of dust-limitation devices be weighed against potential benefits. The textile industry has been pressing for such a change in the U.S. Supreme Court, as a way of liberalizing what it regards as an excessively expensive set of requirements.

The cotton dust standard is one of the most important and controversial the administration has attacked in its determination to ease the regulatory burden on business.

If the regulation proposal is fully implemented, it could end the Supreme Court case, considered one of the most significant in the recent history of regulatory law. The justices themselves would have to decide whether the rule-change renders the controversial case moot or in need of reconsideration in the lower courts.

The cost-benefit sure arose during the Carter administration, not just in the textile industry but in every industry regulated by OSHA. Manufacturers contend that OSHA's standards for exposure to hazardous substances are too harsh and threaten to put small firms out of business.

Organized labor and other textile worker groups consider the use of cost-benefit analysis "immoral" because they say it places a dollar value on human life. They also say it could result in the endless dragging-out of rule-making proceedings with disputes about imponderables: how much standards cost, how many cases of a disease might be prevented and whether the prevention is worth it.

Cotton dust is produced in textile mills as the cotton from the fields is cleaned and combed. If it believed to come not from the fluffy white part of the cotton boll but from leaf and stem material swept up by mechanized cotton-picking machines.

Textile workers say it causes byssinosis or brown lung disease, an emphysema-like respiratory ailment that results in a progressive narrowing of the air passages, coughing, wheezing, fatigue, possible incapacitation and possible death through related heart problems.Many industry experts dispute both the cause of the disease and the plaintiffs' contentions as to its high incidence.

The cotton dust standards in dispute were proposed four years ago by OSHA. They require the installation of numerous vacuum-type machines to suck up the dust and, in areas of highest exposure where standards cannot be met, the use of face-covering respirator masks.

The textile industry challenged the standards as too harsh and too costly. But U.S. Court of Appeals Judge David L. Bazelon ruled against the companies, saying that the law setting up OSHA did not require cost-benefit analysis or consideration of costs to industry. As long as an entire industry was not threatened, he said, the regulations were not too costly.

The Supreme Court accepted the industry appeal and heard oral arguments on it this term. An opinion was expected in the next few months.

OSHA officials were unavailable for comment yesterday on the proposed rule-change. But sources said the solicitor general's office, which was arguing the case before the court, has been informed of it though not its timing.