The Environmental Protection Agency yesterday scrapped standards it had proposed last year for limiting radiation levels around phosphorus mines, weapons plants and federal nuclear reactors, calling the risks "relatively trivial" because so few people live around the sites.

The decision was immediately challenged by environmentalists, who said they will seek to have EPA Administrator William D. Ruckelshaus declared in contempt for violating a federal judge's 1981 order to regulate airborne radiation.

"This decision is illegal and the policies underlying it are wrong," said David Doniger, a senior attorney for the Natural Resources Defense Council. "Exposed people in sparsely populated places deserve protection just as much as those living in big cities -- the protection you get from EPA shouldn't depend on how many neighbors you have."

Assistant EPA Administrator Joseph A. Cannon acknowledged that the decision "does represent a change in policy. There's no question about that." But he and other officials defended the decision as part of the agency's attempt to assess environmental health risks and determine which are "significant" enough to be regulated.

"I don't think it represents a retreat from anything," Cannon said. "We are deciding what risks are big enough to go after."

The EPA, acting under the 1981 court order, issued proposed standards for airborne radioactive materials, called radionuclides, in April 1983. Under the proposed standards, persons living near phosphorus plants, federal nuclear and weapons facilities and underground uranium mines would generally have been exposed to no more radiation each year than they would get in a typical chest X-ray.

Yesterday, however, under the gun of another court order to make its decision final, the EPA withdrew the standards. Cannon said the agency had decided that the risks are "relatively trivial," even though the agency's risk assessments estimate that current emissions could result in one cancer death for every 1,000 persons exposed for a lifetime.

The EPA said that because of the relatively light population around the sites, most of which are in western states, the risks translate into fewer than one cancer death every 13 years.

The agency said, however, it will develop new regulations for underground uranium mines, where risks run as high as one in 100. According to the agency's calculations, that translates into five deaths a year.

Cannon said the EPA had withdrawn its original rules for uranium mines because they were illegal under the Clean Air Act. The agency had suggested that mines could either install ventilating fans and tall stacks to disperse the radiation or could simply buy enough land to keep people far away from the mine.

"Such approaches to pollution control are generally forbidden by the act," Cannon said yesterday. "You're just spreading around the risk."

According to environmentalists, however, the action EPA took yesterday also is forbidden by the law. The Clean Air Act states that the public must be protected "with an ample margin of safety" against any hazardous pollutant that "may reasonably be anticipated to result in an increase in mortality."

EPA officials have argued that the language is impossible to meet. "We are convinced, and I believe Congress is convinced, that that's not what they meant," Cannon said yesterday.

In a letter this week to Ruckelshaus, however, three members of the Senate Environment and Public Works Committee disagreed. The letter, signed by Chairman Robert T. Stafford (R-Vt.), and Democrats Jennings Randolph (W.Va.) and George J. Mitchell (Maine) said they were "hard pressed to understand" the decision.

In background documents, the EPA also said that it had considered "the high cost of controls versus the public health benefits." The clean air law expressly forbids use of cost-benefit analysis in setting standards to protect public health.