There is no escaping the rainbows at Rosemary Hills Primary School in Silver Spring.

A crescent-shaped garden of tulips and pansies lies at the foot of the flagpole. Painted rainbows adorn the walls of the library, classrooms and the principal's office. At the front desk, any child with $5.50 can buy a T-shirt proclaiming him or her a "Rosemary Rainbow."

The symbolism, educators had hoped, would be grasped immediately even by the kindergarten intellect: The diverse races and cultures at the single-story brick school blend together like the hues of a rainbow.

Rosemary Hills, just off East-West Highway, is the centerpiece of voluntary efforts to desegregate Montgomery County public schools. Five years after the program began, Principal Drucille Stafford and many parents cite the school's 300 children as proof that education can thrive in a muti-racial environment.

But among some parents in Chevy Chase and Kensington -- preciselyl how many is unclear -- a grapevine of rumor and concern has taken root. Transfers to private or other public schools have helped push white enrollment at the school down to 45 percent or lower, compared to the 58 percent the plan had projected.

These transfers figured in a threat by officials of the U.S. Department of Education earlier this year to deny the school system a requested $500,000 grant, and in a complaint filed by parents with the U.S. Department of Education's Office for Civil Rights alleging that resegregation was under way at Rosemary Hills.

Indeed, the controversy around the school demonstrates that a classic dilemma of urban middle-class mothers and fathers --- whether to entrust their children to public schools -- has found its way to the suburbs. And because the county's school system helped draw many families from homes in the District, emotions have run high.

On telephones and in pool-side conversations at the Chevy Chase Recreation Center, parents have traded endless stories about the school and six others that make up the Rosemary Hills cluster. Newcomers to the area quickly hear the stories from neighbors and friends. They hear that "tough" children harass others on the playground and run wild in the hall, that the bus rides are too long, pupils at times regress intellectually, and administrators are more concerned with politics than education. Below the surface of some of these complaints is the insinuation that the problem is too many black children from low-income homes in Silver Spring.

As a result, it appears, many parents pulled their children out. "They thought it was too tough, just too chaotic," says Chevy Chase resident Pat O'Rourke, who has two children at Rosemary Hills and gives it high marks as a school. The concern crossed racial lines, parents say: some middle-class blacks were reluctant to have their children exposed to ghetto ways.

The Rosemary Hills cluster, one of three such groupings in the county, grew out of complaints in the early 1970s that as black and Hispanic families moved into Silver Spring, the school's student body had shifted to virtually 100 percent of these minority groups.

In 1976 the school board, acting voluntarily, linked the schools into a cluster and brought into use the county's yellow buses. Black and Hispanic children at Rosemary Hills were dispersed to other cluster schools, with children from white-dominated schools being brought into Rosemary Hills.

Rosemary Hills would offer only kindergarten through 2nd grade. For 3rd through 6th grades, children would go to nearly Chevy Chase Elementary School in one of the most expensive neighborhoods in Montgomery County. Other cluster schools would absorb minority children and provide special programs such as "immersion" in Spanish or freely structured "open classrooms."

Meanwhile, extra teachers, aides, counselers and federal Title I funds at Rosemary Hills would overcome the challenges of educating children of diverse economic and social backgrounds in single classrooms. The result, it was hoped, would be the county's stated goal of "Quality Integrated Education."

The white proportion at Rosemary Hills was projected to rise to 58 percent by the current school year. But the projections were wrong. In 1977, the white proportion was 52 percent and falling. For the school year just ending, officials provide varying figures of 45 and 42 percent, making white a minority again. School officials disagree as to whether the white percentage will continue to drop for the term beginning next September.

The minority population is increasing in the school system as a whole, up from 9 percent a decade ago to about 22 percent today. At Rosemary Hills, the process has been accelerated by transfers of individual students.

Figures are not available on how many have left for private schools.Transfers to other county schools, however, are known to have resulted in a net loss of 22 whites, compared to 15 minority students, according to Loretta Webb, director of the county's Quality Integrated Education office.

Most transfers were granted because parents said their children were not prepared for the full-day kindergarten offered at Rosemary Hills, according to principal Stafford. Other moves approved for parents who wanted their children to have Spanish language instruction at Rock Creek Forest or free-structured teaching at North Chevy Chase Elementary School.

Each of these schools had far fewer minority children than Rosemary Hills.

A Chevy Chase lawyer with a daughter at Rosemary Hills says that neighbors on one side of his home have a girl in a Bethesda school, while neighbors on the other side have a girl in a Jewish parochial school. A family across the street has enrolled a girl in a private school.

Last fall, one group of parents filed a complaint at the federal Office of Civil Rights alleging that liberal school board policy on such things as transfers was fostering resegregation at Rosemary Hills. Federal officials investigated the complaint this winter and a decision is pending.

In addition, the U.S. Department of Education earlier this year informed the county that it was ineligible for a $500,000 grant it had sought under the Emergency School Assistance Act to help finance desegregation. The county's transfer policy and continuing minority concentration were cited. The school board later rebutted the department's contentions and is now awaiting a decision.

Principal Stafford, a former education assistant to HEW Secretary Joseph Califano, says her school has a very low teacher-pupil ratio, about 20:1, and every teacher is there by choice. It also has teaching aides in classrooms, a student counseler, a curriculum coordinator and other specialist positions not found at ordinary county schools.

Having extra staff gets results, says Dr. Marcia Wilson, the curriculum coordinator. "Of 104 first-graders, we only have eight who are not reading at this time. That's extraordinary . . . ." A majority of the school's student's are performing at or above normal levels of competence for their grade level, Dr. Wilson says.

Stafford insists that visitors look over her school, where on a recent day white, Asian, black and Hispanic children watched a film and sat on carpeted classroom floors listening to teachers' stories. In discipline, the school faces only the problems of any primary school, she said.

Some school activities, she explains, are designed to foster racial tolerance and understanding. In March this year, for example, the school staged "Rainbow Week," focusing on a different ethnic group each day. Parents were invited to come in to cook ethnic food.

She points out the rainbow garden beneath the flagpole, planted by students last fall and jealously guarded from vandalism. Brightly colored artwork hangs from walls and blackboards. "These children don't know how to paint anything without putting a rainbow in it," she jokes.

Many Rosemary Hills parents cherish the school's diversity. Explains Judith Siegel, a Chevy Chase mother who has a daughter in kindergarten at the school: "For a short bus ride she gets a very rich and varied educational experience . . . My children do not seem to make racial distinctions when talking about their classmates."

When she moved to the neighborhood last year, she said, she heard vague stories from other parents about poor teaching and discipline at the school. "But all of them turned out not to be true," she says. Her daughter has become attached to the school, refusing any suggestion of going elsewhere with the words, "I'm a Rosemary Rainbow."

But some parents see it differently. They object to busing when there is a school down the block. They feel teaching quality is spotty and in any case think teachers face an impossible task in trying to teach low-income children and wealthy ones in the same room.

Norman Blumenthal, a federal attorney with a daughter in the first grade at the school, feels politics has been put before education. "I object to making 5-year-olds the grunts [foot-soldiers] in an intellectual, ideological war over appropriate ways to achieve desegregation," he says. It is unrealistic to try to mix children who go on ski vacations in Utah and children from subsidized housing in a single classroom, he suggests. They have too little in common. "I'm not sure it's fostering any positive attitudes," he says. ". . . It's just a cheap way to express compassion."

Blumenthal also expresses concern over what he sees as lack of tolerance to criticism in the neighborhoods. "You oppose busing and you're a right-wing nut." Meanwhile, he says, large numbers of parents remain silent and slip their children out through the transfer loopholes.

Many parents believe that for the next school year officials have tightened up on transfers, though Dr. Alan Dodd, the associate superintendent in charge of the school, will say only that officials are "monitoring" requests more closely. The school plans to offer half-day kindergarten, cited by so many parents seeking transfers last year. Curriculum director Wilson notes with irony that no one has signed up for it.

County policy calls for action whenever minority enrollment passes over 50 percent in a given school. At Rosemary Hills, reduction is unlikely until the 1982-83 school year. The board's 15-year plan calls for the closure of two nearby schools, Lynbrook and Rollingwood. Rosemary Hills and other cluster schools would absorb those students, Dodd said, pushing white percentages up to about 58 percent.