A judge ruled yesterday that a county zoning law that defines a family as a group of four or fewer persons, unless they are related by blood or marriage, has unconstitutionally prevented a couple here from operating a group home for problem children in a residential area.

The ruling by Circuit Court Judge George B. Rasin applies only in Caroline County, but can be cited as precedent if challenges to similar local zoning laws arise elsewhere. Courts in New York, New Jersey, Wisconsin and Illinois already have struck down other laws defining a "family," and the city of Manassas, Va., is being sued for its restrictions that rule out a group home for mentally retarded adults in a residential neighborhood.

In addition to finding that the zoning law violated the U.S. Constitution's guarantee of due process, Judge Rasin also said the law, which makes no provision for the operation of juvenile group homes, also violates the state's intent to have such facilities for troubled children.

In a sharply worded ruling from the bench, Rasin said, "our society has certainly advanced to the point where we try to treat people fairly, particularly children. I see no purpose for a jurisdiction to say we're not going to treat our own problem children, but send them somewhere else."

He cited a recent opinion by Maryland Attorney General Francis B. Burch dealing with a group home in Prince George's County that said that since the General Assembly mandated group homes by enacting laws creating them, the county can regulate only health and safety in the homes.

The challenge to the zoning law came from Ricky and Bea Loscomb, a couple with three children of their own. They had operated a juvenile home in East New Market, a town near Cambridge, for about three years before moving last October to Bethlehem, a town with a population of 100 in Caroline County.

At the time of their move, they were caring for six children, age 13 to 17, for the State Department of Juvenile Services. The Loscombs earn their living by providing extensive services for the troubled children. The state pays them $657 per month for each child placed in their home.

When the Loscombs asked Caroline County for a zoning permit to continue operating their group home, called Way Homes, Inc., in their 100-year-old white frame house, neighbors protested and the permit was denied.

Nineteen Bethlehem residents opposed the Loscomb's request, saying they felt their homes would be broken into, property values would decrease, community standards would be lowered, and their farm equipment would not be safe "with these children around."

"The people in the community are nice people," said Ricky Loscomb, 32, "but group homes are new to everybody and there are a lot of misconceptions about them." The state stopped sending the Loscombs more children until the issues were settled in court.

With yesterday's ruling, the Loscombs said they are anxious to welcome eight more children into their four-bedroom home.

In arguing the current court case, Caroline County officials noted that a 1974 Supreme Court decision said a local government can make zoning laws for state sponsored facilities as long as a "resonable purpose" is served by the local zoning, according to Caroline County Attorney Robert E. Jarrell.

But the judge said, "If a group is no larger than a normal family, living together as a family unit . . . I do not believe the zoning ordinance can go behind the relationships. Dormitories and fraternity houses and such are families."

"It's interesting to me," Rasin added, that over the years zoing ordinances for new towns, such as Columbia, have never considered land for such things as jails, mental hospitals and facilities for the retarded."