The Nuclear Regulatory Commission's refusal to consider the most dangerous class of nuclear accidents in licensing new reactors faces a challenge in the Supreme Court as well as strong internal criticism from the agency staff.

In the court, the issue is whether the commission policy, adopted in 1971, can be squared with the 1969 National Environmental Policy Act's requirement that the NRC give "appropriate" consideration to the environmental impact of its major actioons.

Last December, the U.S. Court of Appeals here ruled for the commission. "It is well settled that, because of the extreme improbability of their occurence, the NRC need not consider the environmental effects of so-called 'class 9' accidents," the court held. A "class 9 accident" is one considered to be "severe" or "catastrophic."

In January, two months before the class 9 accident at Three Mile Island, the appeals court voted 9 to 0 to deny a rehearing on the issue. But its unanimity about the "extreme improbability" of a class 9 accident turns out not to have been shared by the staff of the agency and its predecessor, the Atomic Energy Commission. ,tIn 1976, for example, Jimmy Carter's campaign policy committee was warned 'that sooner or later a major disaster will occur at a nuclear generating facility." The warning was in a private letter from John F. O'Leary, a former AEC staff official who served until recently as deputy energy secretary.

His letter was disclosed last May by The New York Times. A few weeks ago, The Times reported that an NRC task force had urged the commission to consider the possibility of a major accident when it develops population guidelines for new reactor locations.

But the staff recommended that the possibility of a major accident be ommitted from consideration in the licensing of specific reactors. That recommendation was criticized in a memo to the task force from the commission's executive legal director, Howard K. Shapar.

He wrote "that any refusal to look at class 9 accidents or residual risks on a site specific basis will provoke substantial controversy and give rise to the implication that the commission is not interested in full disclosure of reactor risks to the people who may be affected by them."

In the case now before the Supreme Court, a group of Florida residents seeks review of the appellate court's decision upholding the commission's granting of a construction license to Florida Power & Light Co. for a reactor at St. Lucie on Hutchinson Island. The site, near Fort Pierce on the state's east coast, is in a region popular with tourists and occasionally hit by hurricanes.

The dispute dates back to 1971, when the AEC proposed to consider, through a rule-making procedure, the impact of possible nuclear accidents. The plan was to divide accidents into eight classes of environmental impact. Each class was one foreseen by the agency and reflected in its design and engineering requirements.

Class 9 was reserved for all accidents that were unforeseen and for which there were no preventive requirements. As a result, license applicants would not be asked to discuss -- and the commission wouldn't consider -- class 9 accidents. The rationale was the same as the appeals court's: "the probability of their occurrence is so small that their environmental risk is extremely low . . .."

The commission never formally adopted the rule. Instead, it declared the rule to be "interim policy" and followed it as if it had been adopted.

Under the interim policy, the commission rejected challenges to the licensing of the Florida plant and approved the plant's license two years ago.

The Florida litigants, represented by Martin H. Hodder and law Prof. Terence J. Anderson, both of Miami, argue in their Etition for Supreme Court review that the AEC hadn't claimed to have a factual basis for concluding in 1971 that it could rationally calculate the improbability of class 9 accidents.

They also recalled the NRC'S repudiation last January of an assumption -- made in the draft of a 1974 report commissioned by the AEC -- that the possibility of class 9 accidents was highly remote.

"The events at Three Mile Island dramatically corroborated the commission's belated concern" and "have justly shaken the public's confidence" in its procedures, Hodder and Anderson wrote.

They acknowledged that the NRC may be correct in assessing a major nuclear accident as extremely unlikely. Nevertheless, the contended, the agency was bound by the environmental law and the Administrative Procedure Act to consider the environmental impact of such an accident.

They also protested that it was improper for the agency to refuse to consider the impact of such an accident unless an intervenor shows "a reasonable possibility" that one will occur.

By contrast, Solicitor General Wade H. McCree Jr. argued for the commission that the Supreme Court has held repeatedly that federal agencies have discrettion to proceed as the NRC did.