In a court test that could affect several of the nation's financially strapped city school systems, a Boston teachers' union is appealing to the Supreme Court a desegregation order that has forced the layoff of 1,100 white teachers in violation of contract seniority rights.

The case already is being called the next reverse discrimination test. A federal judge allowed the Boston school board to fire whites first in a financial bind. Black teachers were retained to meet a racial balance requirement.

American Federation of Teachers President Albert Shanker said last week that if the lower court opinions are upheld "it will be the first time in U.S. history that a federal court, acting in the name of the Constitution, has ruled that people must lose their jobs on the basis of race."

Lower court decisions in Buffalo, Kalamazoo, Mich., and Springfield, Ill., affecting union locals represented by the rival National Education Association, also pit teacher contracts against minorities and have resulted in whites-first layoffs, according to AFT attorney Bruce A. Miller.

The same issue could someday affect other large school districts, including Chicago, Dallas and Philadelphia, which are under desegregation plans that include teacher transfer agreements. With falling student enrollments and dwindling financial resources, such inner city school districts could face layoffs determined by race.

The Reagan administration opposes quota systems in affirmative action programs, and Justice Department officials are studying the Boston case with an eye toward backing the AFT position. One Justice official said it could become "the next Bakke," the landmark case in which a white student won admission to medical school after challenging an affirmative action plan.

Shanker said the Bakke case was weaker than the Boston appeal because Bakke wasn't guaranteed a place in medical school like the teachers' contract guaranteed seniority rights.

The Boston case grew out of a 1975 desegregation order in which U.S. District Court Judge W. Arthur Garrity Jr. ruled that the teaching staff must reflect the city's 20 percent black population.

Last year the school board announced it had to lay off several hundred teachers for financial reasons. To keep the racial balance, it laid off experienced white teachers instead of newly hired blacks.

A few weeks ago, the board announced it would have to lay off another 595 teachers by the end of the month, and so far only white teachers have received dismissal notices, the union said.

Similar race-conscious layoffs have hit Kalamazoo. Pete McNenly, director of legal services for the Michigan Education Association, said his group is appealing the layoffs.

But the issue is more sensitive at his union's national headquarters. John Cox, NEA's director of human and civil rights, said NEA is not prepared to say it would support a local defense of a contract that conflicts with a desegregation order. "They are tough calls," he said. "I can't guarantee which way we'd come out."

In New York City, desegregation of the teaching staff is also an issue. Department of Education officials and the city school board are fighting over whether teachers have to be transferred involuntarily to improve racial balance in its 55,000-member teaching force.

But the New York agreement came after a Department of Education complaint. There was no court finding of discrimination, as there was in Boston. Garrity's desegregation order was followed by a drop in white enrollment in Boston public schools from 70 percent to 33 percent.

AFT's position in Boston is that the white teachers are not responsible for the discrimination and should not have to bear the burden.

In a news conference announcing the appeal last week, Shanker said, "The national AFT has decided to become involved in this case because we believe that colorblind, race-neutral seniority systems are perhaps the most important safeguard won by the American labor movement in its 100 years of struggle for job equality for all Americans."

The A. Philip Randolph Institute, a group of black trade unionists, also supports the union position.

So far, the courts have not. The 1st U.S. Circuit Court of Appeals rejected the union's arguments last February. It said the victims in the Boston case were the city's black children and noted "the elimination of the vestiges of a segregated school system cannot be accomplished until the effects of past hiring discrimination have been eradicated."

An integrated faculty, the court said, is necessary to bring black students and parents "fully into the school community and decision making process and to counteract their past isolation."

Race-conscious remedies, such as laying off white teachers first, are permitted "where colorblind approaches would be inadequate," the appeals court said. The Boston school board had said its percentage of black teachers would have dropped from 19 percent to 8 percent with the layoffs, unless senior white teachers went first.