The Reagan administration was preparing yesterday to enter a major "reverse discrimination" case at the Supreme Court on the side of white firefighters and police in Boston who are challenging an affirmative-action court order.

The affirmative-action order, now the center of one of the country's biggest civil rights controversies, required that whites be laid off during municipal budget cuts rather than blacks and Hispanics with less seniority.

Although this violates the state's "last hired, first fired" seniority law for public employes, it was done to maintain minority hiring percentages in Boston's police and fire departments imposed by a federal court after the court ruled that the city had discriminated.

The dispute raises for the first time at the court the question of whether the Constitution permits whites to be removed from jobs, even temporarily, because of their race -- because if they had been black or Hispanic they would have remained employed.

The case has taken on added importance for unions and state and local governments because of recent layoffs and reductions in force.

Sources said Solicitor General Rex E. Lee will submit a "friend of the court" brief arguing that the lower courts overstepped their authority. It would be the first time that the U.S. government has come in on the side of whites in a major Supreme Court reverse discrimination case. It also would complete a nearly across-the-board revision of government litigation policy on several sensitive constitutional issues -- such as tax exemptions for discriminatory schools, busing and abortion -- that began shortly after Reagan took office.

The court agreed to review the Boston cases (Boston Firefighters Union vs. Boston Chapter NAACP, et al.) on Nov. 1. The controversy stemmed from successful discrimination complaints brought by blacks, Hispanics and the NAACP against recruitment policies and hiring examinations in Massachusetts.

A U.S. District Court held in 1974 that "racial discrimination had led to the virtual exclusion" of blacks and Hispanics and ordered remedial actions that brought dramatic increases in minority employment: from 2.3 percent in the police department to 11.7 percent, and from less than 1 percent in the fire department to 14.7 percent. Boston is now 30 percent black and Hispanic.

These gains were threatened in 1981 by Proposition 2 1/2, a budget-cutting referendum that produced massive layoffs. Because the recently hired minorities lacked seniority over whites, layoffs by seniority would have caused the departments to "regress to the state of precipitous racial imbalance" that originally prevailed, the 1st U.S. Circuit Court of Appeals ruled last May. Police and firefighter unions and the state of Massachusetts appealed to the Supreme Court.

Several hundred whites were laid off under the ruling and lost salaries during the period of the layoffs. Though they have since been reinstated, they say it could happen again.

The Reagan administration's position in the case is no surprise. Attorney General William French Smith and other officials have said in numerous forums that they favor "color blind" employment policies and generally oppose quotas.

But since the government is not a party to the controversy in court, it has a choice of whether to express its views to the justices, as the Carter administration did in the Bakke reverse discrimination case.

The Massachusetts dispute is different from previous affirmative-action cases. The Bakke case in 1978 involved denial of admission to a medical school, not expulsion or suspension of a student. The Weber case in 1979 concerned private affirmative action undertaken by an employer and a union, not the discharge or laying off of white workers and replacement by minorities.