The Supreme Court yesterday backed out of the major affirmative action case it had previously agreed to decide, ending the Reagan administration's hope for a strong ruling against "reverse discrimination" this year.

The court, without dissent, said it would not rule on a challenge by whites to an order protecting blacks and Hispanics from layoffs in the Boston police and fire departments. The court indicated that it believed the dispute is moot because the layoffs have been rescinded.

The action, while leaving the issue unresolved, was a victory for the NAACP and other civil rights groups that feared the court might agree with the Reagan administration. It had chosen the case to make its first stand in the Supreme Court against major affirmative-action orders. It was the second time in three years that the court has changed its mind about ruling on affirmative action, leaving lower courts free to do as they please in this sensitive field.

The Boston layoff order was issued in 1981 to preserve jobs for blacks and Hispanics during voter-mandated budget cuts. Blacks and Hispanics had been employed only a short time in the police and fire departments. Under Massachusetts' "last-hired, first-fired" seniority law, they were to be the first laid off.

But a U.S. District Court judge, saying the layoffs could wipe out dramatic gains in minority employment achieved under an earlier court order, told the city to ignore the state seniority law and to protect those gains. The 1st U.S. Circuit Court of Appeals upheld the ruling.

As a result, whites with greater seniority bore the brunt of the layoffs. They and their unions appealed to the Supreme Court. In 1982, however, Massachusetts enacted legislation giving the city more money and requiring that all those laid off be reinstated. The justices cited that yesterday as they nullified the Court of Appeals ruling in Boston Firefighters Union, Local 718, vs. Boston Chapter, NAACP, and told the lower court to consider whether controversy remained in the case.

In another action yesterday, the justices upheld the system used by the government to help settle disputes over eligibility for Social Security disability benefits.

The benefits go to insured people unable to continue work for health reasons. If denied payments because they are considered employable, applicants may appeal to an administrative law judge. These disputes, hundreds of thousands each year, often are settled in hearings with the help of elaborate medical-vocational guidelines, including job market tables or "grids" devised by the government in 1978 to show whether employment exists for people with certain kinds of disabilities.

The 2nd U.S. Circuit Court of Appeals, ruling in the case of a 51-year-old woman with a back condition and hypertension, said that applicants deserved a more individualized determination of employability than that provided by the generalized tables.

In an 8-to-1 vote yesterday, with Justice Byron Raymond White dissenting, the justices reversed the lower court. Justice Lewis F. Powell Jr., writing for the majority in Heckler vs. Carmen Campbell, said the guidelines conformed with the Social Security Act and are an efficient way of determining the existence of jobs in the national economy. To require individualized proof at each hearing "would hinder needlessly an already overburdened agency," he said.

Carmen Campbell, the applicant in yesterday's case, was a poorly educated former hotel maid who spoke limited English. But she did not have a lawyer in her hearing administrative law hearing. In an unusual aside in yesterday's case, Justice William J. Brennan Jr. sharply criticized the administrative law judge for not helping Campbell understand what was going on and defend her claim.

He said the judges have a legal "duty" to provide such assistance. "The federal courts have been successful in enforcing this duty in the past," Brennan said in a concurring statement, "and I respectfully suggest that the secretary of health and human services insist upon its faithful performance in future cases."

In other action yesterday, the court agreed to consider a lower court ruling temporarily blocking Secretary of Interior James G. Watt's plans to sell oil and gas leases off the coast of California. The justices are expected to rule next year in Watt vs. California.