The Los Angeles Unified School District, second-largest in the country, has ordered that high school teams in a special academic competition be racially and sexually balanced or face disqualification.

The order, which has enraged many white parents and students in a system long troubled by desegregation disputes, came after an all-white team with only one female member won last year's citywide "academic decathlon."

The national competition tests students in subjects such as mathematics, fine arts, economics and English.

Rose Gilbert, the English teacher who coaches the championship team from Palisades High School, said she was "outraged" by the order because "it is based on a quota system and not intellectual strength."

She questioned the district's insistence that her team "reasonably" reflect the school's ethnic makeup when the same is not required of the school's athletic teams. She said that minorities have participated in previous years, but that the two leading minority prospects this year declined to compete "because it was too much pressure."

The school is located in the affluent Pacific Palisades neighborhood where President Reagan maintained a home before his election in 1980. Busing to achieve racial balance has produced a student body that is about half white, 42 percent black, 4 percent Latino and 4 percent Asian.

Gilbert noted that the district has not complained about the basketball team being 75 percent black and the football team about 80 percent black.

"Athletic competition primarily is a question of physical skill, which is somewhat more inherent than intellectual skill," said Marty Estrin, a spokesman for the school district.

School board member Roberta Weintraub, representing the city's predominantly white northwest area, said, "I don't like the whole idea of a quota system. It's absolutely ridiculous."

The academic decathlon competition was created in 1966 by Robert Peterson, now school superintendent for neighboring Orange County. It has spread to 33 other states, including occasional competitions in Maryland, Virginia and the District. It is unusual among academic competitions because it requires each team to be composed of two "A" students, two "B" students and two students with a grade average of "C" or lower.

Although the rules do not call for ethnic or sexual balance, Peterson said he favors the Los Angeles action if it "encourages a whole new category of student to be part of the program."

Paul Possemato, director of the Los Angeles schools senior high division, said he had been concerned in past years about academic teams not being "symbolic" of the educational process at their schools. He said he had not received any formal complaint about the Palisades teams, which have won the city competition four years straight.

He said each academic coach will be required to report the ethnic and sexual makeup of their teams a month before the November competition and will be told if changes must be made.

"I'm going to do a visual review on the day of the competition, and teams which don't comply will not be permitted to compete," he said.

Possemato said he does not feel his decision had anything to do with the racial makeup of athletic teams. Rita Walters, a school board member who is black, said she supports the action and thinks Palisades coaches should do more to attract whites to basketball.