The Justice Department asked a federal appeals court yesterday to allow reconsideration of a school desegregation busing plan in Baton Rouge, La., on the grounds that white students are fleeing the system.

The filing was the first time the Reagan administration has asked an appeals court to restrict a busing plan already in effect, according to J. Harvie Wilkinson III, deputy assistant attorney general for civil rights and one of authors of the motion.

It is also significant because until now the government has rejected school board arguments that "white flight" was a justifiable reason not to use mandatory busing to integrate schools.

The measure of any desegregation plan is its effectiveness, the Justice brief said. "A plan does not work if its disruptive and compulsory features cause large numbers of parents and students to abandon the public school system."

If accepted by the courts, the new federal stance could lead to reopening of school desegregation cases in several major cites, including Boston, Cleveland, Memphis, Detroit and Denver, Justice officials said.

So far, federal courts have rejected such attempts. In 1971, in U.S. v. Scotland Neck Board of Education, the Supreme Court said white flight may be a cause of concern but "cannot be accepted as a reason for achieving anything less than a complete uprooting of the dual public school system."

A few weeks ago, the Sixth U.S. Circuit Court of Appeals used similar reasoning in rejecting a district court desegregation plan in Nashville.

Civil rights attorneys outside the government were distressed at this latest turnabout by the Reagan administration. William Taylor of the Leadership Conference of Civil Rights said the government "seems to trying to turn previous rulings on their head . . . . They're using the prospect of white flight to suggest we should go back to more segregation."

William Bradford Reynolds, head of the department's civil rights division, said at the American Bar Association convention in San Francisco yesterday that "Our view is that the order that the lower court put into effect in Baton Rouge is not working and it's indicated that there's a need to modify it. We think that the thing for the court of appeals to do is stay its hand until everybody gets a chance to go back to the district court and take a hard look at what is really effective in the way of a desegregation remedy."

In the Louisiana case, the East Baton Rouge Parish School Board was first sued in 1956 by black parents who contended that the school system was unconstitutionally segregated. A federal district court agreed and over the years various attempts were made to wipe out the unconstitutional segregation.

None, however, worked, according to the court, and last fall the system was ordered to institute busing. The school board took its case to the Fifth Circuit Court of Appeals. Meanwhile, busing began, and the school board says that 4,000 of the 35,000 students in its elementary schools have left the system since then.

The federal government entered the case in 1979 on the side of the black parents, arguing for busing. School board attorney John F. Ward Jr. said yesterday that the government has switched sides, siding for the first time with the school board.

In its motion yesterday asking the appeals court to delay consideration of the case, Justice attorneys acknowledged that Carter administration officials had argued for busing, but the new administration has reconsidered the usefulness of busing, they said. "Experience has shown court-ordered transportation generally to be a failed experiment."

Justice attorneys emphasized in their filing that they agree that the Baton Rouge district is guilty of violating the Constitution by maintaining a segregated schools system. The court record shows that more than 60 percent of the schools in the district are one-race schools. About 27,000 of the 60,000 students in the East Baton Rouge Parish system are black.

The Justice motion also noted that the latest school board effort to come up with an alternative to the district court's busing plan was faulty. Its proposal to use "magnet schools" was "an empty shell--with not even a suggestion of adequate funding, staffing or recruitment essential to lay the foundation for success," it said.

But they added that the government wants to work "to help devise a desegregative package that will remedy the violation properly found to exist in East Baton Rouge without resort to compulsory transportation of students."

Wilkinson, who joined the department a month ago from the University of Virginia Law School, said, "Our position is that you can have effective desegregation built on community support. Offering unique educational opportunties can attract people voluntarily."

In the motion, Justice said, "Those who advocated this remedy busing as the cure-all for segregation in the classroom badly miscalculated community response. In our view, that dimension of the school desegregation effort can no longer be so lightly dismissed if we are continue to be faithful--as we must--to the Supreme Court's command . . . that we find 'effective' desegregation remedies, and find them 'now.' "

Busing children "who are themselves innocent of any wrongdoing in order to remedy a constitutional violation brought about by the conduct of officials decades earlier is at odds" with the principle that legal burdens should bear some relationship to individual responsibility, the court papers said.

Voluntary incentives "can more effectively dismantle this dual system of long standing than the measures presently in effect," the Justice lawyers argued.

Reynolds made some of the same points against busing in a speech to the ABA yesterday. He said the idea of "racial balance" rather than "racial neutrality" became the overriding concern in desegregation cases after the Supreme Court first approved of busing as a remedy in a Charlotte, N.C., case in 1971.

He alluded to white flight when he said "this preoccupation with mandatory busing has generally produced racial isolation on a broader scale. In case after case, economically able parents have refused to permit their children to travel unnecessary distances to attend public schools, choosing instead to enroll them in private schools or move beyond the reach of the desegregation decree."