The Supreme Court yesterday agreed to consider a $10.5 million damage claim brought by the family of the late Karen Silkwood, who was contaminated by radiation in 1974 while employed by the Kerr-McGee plutonium processing plant.

The outcome could determine whether states and state laws can play a major role in nuclear safety regulation or whether safety breaches must be handled by federal regulatory authorities, as the nuclear industry and the Reagan administration desire.

The 10th U.S. Circuit Court of Appeals threw out most of a jury's award to the Silkwood family made under state laws, saying that nuclear regulation is the business of the federal government.

The Silkwood case was one of the earliest and most celebrated controversies concerning the safety of the nuclear industry. Silkwood was a laboratory analyst in the Cimarron, Okla., plant of Kerr-McGee, which manufactured plutonium pins used as fuel for nuclear reactors.

At age 28 she was contaminated with between 25 and 50 percent of the permissible lifetime limits for exposure. Eventually, the source of the radiation was found to be plutonium in Silkwood's home. While Kerr-McGee alleged that she had put the plutonium there, the jury in the case rejected that contention.

On Nov. 13, 1974, she was killed in a car accident while on her way to meet a reporter to deliver documents alleging falsification of quality-control records at the plant. Her case became a symbol of opposition to the nuclear power industry.

Bill Silkwood, her father, sued Kerr-McGee, asking $10 million in punitive damages for unsafe conditions and careless practices he said contributed to the contamination.

After an 11-week trial in 1979 an Oklahoma federal court jury awarded the money, plus $500,000 as compensation for losses she reputedly had suffered.

The federal appeals court rejected all but $5,000 of the award, however, saying that the regulation of nuclear power was exclusively a federal domain under atomic energy laws and that damage suits brought under state laws would compete with the federal regulatory structure.

Silkwood's lawyers and the 18 states that have become involved in the case say the ruling "leaves the states powerless to meaningfully protect their citizens against reckless and life-threatening mishandling of radioactive materials," as Wisconsin Attorney General Bronson C. LaFollette told the justices in a friend-of-the-court brief. The justices conditionally accepted the case of Silkwood vs. Kerr-McGee Corp. subject to their determination after hearing oral argument that they have jurisdiction.

In other action yesterday:

The court rejected another appeal by Jeffrey R. MacDonald, the former Green Beret physician convicted of murdering his wife and two children at Fort Bragg, N.C., in 1970. MacDonald, serving three consecutive life terms, argued that his conviction should be reversed because the trial judge limited testimony from a witness MacDonald says could have helped exonerate him.

MacDonald said the murderers were hippie-style intruders who invaded his home and attacked him and his family.

Last year the justices sent MacDonald back to prison after reversing an appeals court decision that had freed him because of undue delays in his trial.

The court said it would review the power of federal judges to disqualify defense lawyers from criminal cases because of conflicts of interest. The case, Flanagan vs. U.S., originated with federal charges against four Philadelphia police officers serving on a "grandpop" squad, a decoy unit featuring a detective disguised as a helpless elderly person to attract muggers. The officers were charged with civil rights violations for allegedly making illegal arrests and abusing people on the streets.

The defendants hired a single lawyer, former Philadelphia prosecutor Richard Sprague, to defend them. The judge in the case ruled this was a conflict of interest under legal ethics because the interests of the individual defendants could conflict. The officers appealed to the Supreme Court, arguing that the ruling breached their constitutional right to a lawyer of their choice.

The court agreed to consider a provision of federal racketeering laws that allows forfeiture of businesses and interests involved in corrupt activity. The case, Russello vs. U.S., stems from a successful effort by prosecutors in Florida to seize $340,043 in insurance payments received by a defendant convicted in an arson-for-profit conspiracy.

The court wiped out a lower court decision declaring unconstitutional a congressional amendment designed to keep people out of Comprehensive Employment and Training Act (CETA) programs if they advocate "the violent overthrow" of the government. The case involved a law sponsored by Rep. W.C. (Dan) Daniel (D-Va.) to keep Dorothy Blitz out of the program because she is a member of the Communist Workers Party.

U.S. District Court Judge Barrington Parker said that the law was a violation of constitutional free-speech guarantees. Meanwhile, Blitz had completed the program, eliminating the controversy that the justices yesterday declared moot (Donovan vs. Blitz).