The Supreme Court gave strong hints yesterday that it will not resolve the major "reverse discrimination" case on its docket this year, a challenge to an affirmative-action order protecting minorities in the Boston police and fire departments from layoffs.

Instead, it appeared likely after an hour of oral argument that the justices will declare the case moot because the layoffs have been rescinded.

The 1981 layoff order, designed to preserve jobs for blacks and Hispanics hired after the city was judged to have discriminated in municipal employment, nullifed Massachusetts' "last-hired, first-fired" laws and resulted in whites with greater seniority bearing the brunt of layoffs during voter-mandated budget cuts.

The case had been targeted by the Reagan administration, trade unions and state and local officials as a chance to strike a major blow against affirmative-action orders that penalize whites.

But the justices--in a prolonged public exchange largely among themselves--yesterday repeatedly questioned whether there was anything left for them to rule on, now that the whites who were laid off have all been reinstated.

"Haven't the layoffs all been rescinded?" Justice William J. Brennan Jr. asked Massachusetts Assistant Attorney General Thomas A. Barnico. "What kind of case have we got here?" Brennan said. "Every fireman and police officer has been reinstated. What kind of case have we got left?"

Justices Sandra D. O'Connor and Harry A. Blackmun also wondered whether there was any order currently in effect for the court to rule on.

Only Justice William H. Rehnquist appeared supportive of a ruling on the merits of the case, noting that in a separate proceeding the laid-off employes are seeking back pay for the time they missed on the job.

"There's the back pay issue," Rehnquist interjected.

"But that's not here," responded Blackmun. "That's not before the court."

The discussion of mootness consumed most of Barnico's time yesterday. "My time is expiring," he told the justices as the red light flashed, signaling the end of his 15 minutes. "But my case is not, I assure you," he concluded to the laughter of the court.

John McMahon, representing the police and firefighters' unions in the case, was given an opportunity to discuss the issues.

He said the U.S. District Court order protecting the minorities from layoffs breached federal civil rights law, which protects bona fide seniority systems, and the Constitution.

The rights of the senior workers "were invaded," he said, by minorities who were not themselves the proven victims of racial discrimination in Boston.

"The use of racial classifications should only be approved--if at all--when they are used for the victims of past discrimination," he said, echoing the Reagan administration's position on affirmative action.

James S. Dittmar, the lawyer for the minorities in Boston Firefighters Union et al. vs. Boston Chapter, NAACP et al., said he too thought the case was moot.

If not, he said, the court should uphold the affirmative-action order. "One hallmark of what this court has done throughout the history" of civil rights law, he said, "is to vest lower courts with the flexibility" to remedy discrimination.

While some "third parties" did suffer as a result of the lower court order, Dittmar said such a "burden" is justified to redress proven discrimination.