The Justice Department, departing significantly from the policies of past administrations, yesterday asked the Supreme Court to strike down a controversial "reverse discrimination" order that aided black and Hispanic Boston city employes and resulted in temporary layoffs for hundreds of whites.

The "reverse discrimination" order required that, when it became necessary to cut the municipal budget, white police and firemen be laid off rather than blacks and Hispanics with less seniority.

The purpose was to keep up the level of minorities in the police and fire departments. These percentages were mandated by federal court order after the city was found guilty of discrimination in hiring.

Solicitor General Rex E. Lee, in a "friend of the court" brief, said the lower court ruling brought into being "a new class of victims, completely innocent of any wrongdoing," and tampered with a seniority system in violation of federal civil rights laws.

The brief represents the first time that an administration has intervened on the side of whites in a major reverse discrimination case at the Supreme Court.

Its filing marked the culmination of a substantial debate within the administration. The Equal Employment Opportunities Commission and the U.S. Civil Rights Commission asked the government to side with the minorities in this case, one of the country's most important civil rights controversies.

Public employe union officials immediately praised the administration's action.

"We believe in the seniority system," said Phyllis Franck, speaking for the American Federation of Teachers (AFL-CIO), which recently tried unsuccessfully to challenge a similar order involving teachers in Boston.

"We believe it is designed to protect all employes," she said.

Civil rights lawyers denounced the administration's move. Barry Goldstein, of the NAACP Legal Defense Fund, said the brief, if accepted by the court, could jeopardize all affirmative action powers of the courts to deal not just with layoffs, but employment discrimination generally.

"Not only does the administration not care to propose practical solutions to the historic problems of racial and ethnic discrimination," Goldstein said, "but it is actively seeking to thwart district courts, administrative agencies and others who seek to find practical solutions to the historic problems of discrimination."

The case (Boston Firefighters Union v. Boston Chapter, NAACP, et al) began in 1974, when a federal judge found that the city had discriminated against black and Hispanic applicants to the fire and police departments and ordered hiring and recruitment goals that substantially increased the minority percentages in those departments. In 1981, however, voter-mandated budget cuts prompted heavy layoffs.

Under Massachusetts seniority laws, the brunt of the layoffs would have fallen on the recently hired minorities. To preserve their gains, they successfully sought modification of the seniority system, resulting in whites being laid off rather than blacks and Hispanics with less seniority.

The administration argued yesterday that a provision of the Civil Rights Act's Title VII, the job discrimination law, specifically exempts bona fide seniority systems from interference by federal judges.

As a result of the Boston order, Lee said, "White firefighters and police officers--some with more than 10 years of service--were furloughed, while black and Hispanic employes with as little as two years' seniority were retained. This was not a permissible exercise" of a judge's authority, he said.

The lower courts also sidestepped a provision of the discrimination laws which permits employment decisions without court interference when no discrimination is involved, Lee said. No discrimination was alleged in the seniority system, he argued.

The solicitor general's brief said it was unnecessary for the court to rule on the sensitive constitutional issues involved in the case: whether whites can be removed from jobs because of their race, in order to preserve jobs for blacks and other minorities.

In siding with the whites, the administration rejected recommendations both from the U.S. Civil Rights Commission and the EEOC, which had voted 4 to 1 to urge Lee to come in on the side of the minorities.

This was the administration's position in the case as late as April, 1981. Sources said the agency tried unsuccessfully to insert a footnote in yesterday's brief stating its position.