About a week before Christmas last year, Larry and Lois Kennedy moved into a $120,000 home in Silver Spring's Three Meadows community with the first two of eight mildly retarded persons they hoped to care for in a home-like atmosphere.

Shortly after the new year, while the Kennedys slept, someone rammed the car into their garage and sped away, ominously leaving behind a burning rag. Then, two days later, just about an hour after the Kennedys had turned out their lights and gone to bed, someone fired a shotgun point blank through their front window, splashing glass across the living room and cutting a hole in one wall.

The violence has left not only the Kennedys but many of their neighbours shaken. But those same neighbors, who deny any part in the violence, make clear that they too, are opposed to the locating of a facility for the mildly retarded in the midst of their affluent community.

"I don't have a bias against housing retarded people," said one neighbor, Ruth Rubin, who opposes both the violence and the facility. "But it seems they would be better off in some other neighborhood."

The home, called Jubilee House, has 15 rooms - seven of them bedrooms - and sits on a half-acre plot, according to Larry Kennedy, 34, who has worked for the Lt. Joseph P. Kennedy Institute for the mentally retarded in Northwest Washington.

According to Kennedy, only two mildly retarded individuals, who are employed, now reside in the home. Each pays a monthly fee of about $500 to the Kennedys, who receive no state or federal funding for operating the facility.

"They (the neighbor) didn't understand the kind of people who will be living here," Kennedy said yesterday. "They were wild rumors flying around. They thought there would be people living here who would be dangerous . . . rape children and create havoc in the neighborhood."

But some neighbors such as Rubin say their anger stems more from how the facility came to be in their neighboor in the first place.

THey sneaked it into the neighborhood," said Rubin, who lives about four houses away from the Kennedy residence on Sandy Ridge Road. "Nobody knew about it until the Kennedys were in the house."

Others argue that a facility designed to integrate the mildly retarded into the community as a whole it is poorly located.

"I'm very concerned about the people who are to live in the home." said Elaine Sheehan. "Here , there's no public transportation and there's no way they could walk to a shopping center . . . There's no place on the property for physical or recreational activity and the house just isn't big enough to hold all these people."

At the heart of the neigborhood dispute is opposition to the county ordinance regarding homes for the mentally retarded. In Montgomery, any home used as a facility for the mentally retarded is considered a single family dwelling as long as no more than eight retarded individuals are housed in the facility.

The group homes for the retarded must be licensed by the state and certified by the county, but the citizens are not given a chance to voice their opposition to such a facility in public hearings as they are when group homes for juveniles are proposed.

The Montgomery County Association for Retarded Citizens runs three group homes similar to the Kennedy facility and also shelters retarded individuals in apartments throughout the county. But despite the county's liberal zoning laws in relation to these facilities, there is still shortage of homes and long waiting lists, a spokesman for the association said.

The county does, however, require the individuals opening homes for the retarded to notify neighbor in the immediate vicinity that they plan to move into the community.

Many residents interviewed yesterday said Kennedy had not notified them when he moved into the neighborhood and that they had met him for the first time at a Jan. 6 neighborhood meeting that they had set up with their lawyer and various county officials.

"The community knew we were looking at the house as far back as August," Kennedy said.The home is own by a physician from Potomac.

Police have sworn out a warrant for the arrest of an adult who lives in the community in connection with the arson incident, a police spokesman said yesterday, but no arrest has been made yet.

Meanwhile, Larry Kennedy says he and his wife, who is eight months pregnant, and the home's two residents, live in fear.

Yet despite the attacks on the home and the neighborhood opposition, Kennedy maintained that he bears no ill will towards the community of about 25 homes. He said that about six neighbors have, in fact, come to his home so far to welcome his family.

"We sincerely like being here," he said. "And we see this as opportunity to serve the neighborhood and help the neighbors understand the situation retarded people have.