A group of white schoolteachers, supported by the Reagan administration, asked the Supreme Court yesterday to declare that a voluntary affirmative action plan in Jackson, Mich., is unconstitutional "reverse discrimination."

The case, and two similar ones before the court this year, may finally resolve the controversy between the administration and civil rights groups over the legality of employment programs that favor minorities over whites -- a debate fueled in part by confusion over how to interpret prior court decisions.

The court has held that federal law does not prohibit private employers from setting up voluntary affirmative-action training programs in jobs traditionally closed to blacks. But the court has never said that the Constitution permits public employers to use such programs.

The administration, which filed a friend-of-the-court brief in the cases, is urging the closely divided court to declare that special treatment for minorities can be given only to actual victims of discrimination.

The Jackson case involves the legality of a 1972 contract -- approved by the Jackson teachers' union, which is 80 percent white, and the local school board -- that required layoffs of a disproportionate number of senior white teachers in order to protect recent hiring gains by blacks.

That program was unconstitutional, according to an attorney for the eight white teachers laid off in the last nine years, because it was not based on a judicial finding that it was needed to remedy prior discrimination in Jackson.

The attorney, K. Preston Oade Jr., told the justices that a "race-based system for layoffs" was constitutional "only if strictly necessary to remedy" prior discrimination.

Otherwise, Oade argued, the racial hiring and layoff provisions in this case showed "naked racial preferences" that deny equal protection for white teachers. Such protections cannot be "bargained away" by agreements between the union and school board, he argued.

That view sparked an unusually sharp exchange between Oade and Justice Thurgood Marshall, the court's only black member. "Isn't it true that some of the people you represent got jobs because of their color?" Marshall asked.

Oade replied that he did not know whether that was true.

"Didn't you have segregated schools in Jackson?" Marshall asked. Oade began to reply when Marshall cut him off. "Did you ever have school segregation in Jackson?" Marshall asked.

"Not since 1953," Oade replied.

"I asked you 'ever,' " Marshall said.

"The record [presented in the trial court] only goes back to 1953 and does not show it," Oade said.

"Did you look?" Marshall asked.

"Yes," Oade said.

"And you didn't find it?"

"No."

Jerome Susskind, an attorney for the Jackson school board, said the reason there was no clear proof of prior discrimination is that the board "knew there was discrimination" and undertook a voluntary plan to remedy the problem "without waiting to be sued."

The plan adopted was justified, he argued, because of the need to integrate the schools and to have a diverse faculty.

Susskind said that before 1954 there was "not one minority teacher" among the more than 500 public schoolteachers. By 1972 about 9 percent of the teachers were minorities in a school system that was 16 percent minorities, he said.

"We are in a Catch-22," he argued. "We don't have a good trial court record showing discrimination because we voluntarily integrated our schools. If we'd been recalcitrant and not obeyed this court's rulings and been sued, then we would have a record" on which to justify the race-conscious hiring and layoff plan.

"If you are not going to protect [newly hired minorities] from layoffs," he argued, then the effort would be "just an exercise in futility and a waste of time, and we would be right back where we started."

The court is expected to rule in the case by July.