The Reagan administration has taken its campaign against court-ordered busing to the Supreme Court, asking the justices to give lower courts the flexibility to cancel desegregation plans that cause educational or social disruption, such as white flight.

The objective of school desegregation must sometimes give way when the "educational, social and economic costs" are too great, the administration said in its first attempt to get Supreme Court agreement with the administration's philosophy of school integration.

The government made its anti-busing plea to the high court in a "friend of the court" brief filed Friday in support of Nashville, Tenn., officials seeking a reduction in busing there. Such briefs generally enhance significantly the chance that a review will be granted in a case.

The justices have ducked major reviews of court-mandated busing plans for five years, however, in part because of intense differences among themselves that could further confuse subordinate judges in this sensitive area. The views of Sandra Day O'Connor, the newest justice, could become crucial as the court decides in the coming weeks whether to tackle the issue now.

The Nashville case, Metropolitan County Board of Education of Nashville and Davidson County, Tenn. vs. Kelley, et al, is 26 years old. The most recent chapter was written last year when a U.S. District Court judge ruled that a busing program started in 1971 had failed to desegregate schools adequately while contributing to "white flight" and other disruptions.

The judge eased busing requirements for the city and Davidson County, which are consolidated in a metropolitan government, wiping them out for kindergarten through 12th grades and lowering acceptable percentages of minority enrollment in each school.

The 6th U.S. Circuit Court of Appeals ruled, 2 to 1, for stiffer busing requirements, saying they were required by the Supreme Court's 1971 ruling in Swann vs. Charlotte-Mecklenburg Board of Education that first gave judges the flexibility to order busing.

The Reagan administration believes the Swann decision gave judges an often ignored flexibility not to order busing.

In the government's new friend-of-the-court brief, Solicitor General Rex E. Lee said the appellate court "adopted too restrictive an interpretation" of the Swann case.

Swann did not require busing or a certain racial mix in schools, he said. The court "has never stated" that the objective of desegregation "must be achieved regardless of the costs to individual students or the educational system as a whole," he said.

Swann, he said, "does not preclude district courts from considering competing educational, social and economic costs or from holding, as the district court did, that 'transportation of young children which risks the health of the children or significantly impinges on the educational process' is to be avoided.

"Lower courts, school boards, and governmental enforcement agencies are clearly in need of guidance regarding the proper interpretation of Swann," Lee's brief said.

The administration's arguments have been made in speeches and in the lower courts, most recently in a case involving Baton Rouge, La. The brief marked the first time the administration has formally sought Supreme Court support for its views.

The Reagan anti-busing effort is now in full swing, with the Justice Department studying requests to intervene and undo busing orders in Boston, Cleveland, Detroit, St. Louis and Memphis, among other areas.

Justice Department sources said the Nashville case is an especially convenient vehicle for Supreme Court review of busing, in part because the dissenting appeals court judge, Anthony J. Celebrezze, is regarded as a liberal.

The Supreme Court ruled on two cases last year involving busing. But the issues in Los Angeles and Seattle concerned locally instigated desegregation plans, not plans ordered under the Constitution by federal judges.