The Fairfax County Board of Supervisors refused yesterday to order an investigation of the county School Board's protracted attempt to force the transfer of a 16-year-old retarded girl from a residential institution to a day school.

County Board Chairman John F. Herity's request for a "full investigation" died for lack of a second after several supervisors pointed out that the school board still could decide to appeal the case, which has been in dispute since 1977./

"This case is outside the purview of this board," said Annandale Supervisor Audrey Moore. "It could still be in the courts."

Herrity said in an interview he was not sure if the school board is at fault in the case, but that he wanted to "get to the bottom of it" with an investigation. "To draw something out like this through the courts is not necessarily in the best interests of the county," Herrity said. "There's been enough public expense and public outcry that this issue should be looked at."

A Fairfax judge ruled Thursday that the county may not force the girl, Paula Jane Hurley, to transfer from a private residential institution for the retarded in Ridgely, Md., to a six-hour-a-day public program ner Tysons Corner.

The county, which pays about $8,700 a year for Paula's education, sued her parents in 1980 after having lost twice in reviews by the State Department of Education. The case has cost both sides thousands of dollars. d

In last week's Circuit Court trial, Paul and Rae Hurley, the girl's parents, convinced Judge William E. Plummer that Paula is making good progress at the Benedictine School, where she enrolled shortly after the Hurleys moved to McLean in 1977. They said that Paula, who is severely retarded and cannot speak, has made strides in sign language and controlling her bodily functions at the residential institution. Paula has a mental age of about 3.

The county, on the other hand, argued that its day school could provide Paula with an appropriate education in the least restrictive environment, as state and federal law require. The county said that Paula has not learned to apply the skills learned in the Benedictine school to daily life in the outside world.

Plummer, siding with the Hurleys, ruled tht the county had not proved that it could provide an appropriate education for Paula.

Herrity said yesterday that Dr. Gary Jones, his representative on the school board, plans to conduct a review of the matter himself.

Jones declined to comment on the Hurley case, but said "it is something I want to discuss internally with the school board."

Ann W. Mische, an attorney for the school board, said immediately after the judge's decision that she doubted the county would appeal. Mische was not available for comment yesterday, nor were school officials.

Before the trial, the Hurleys sent Herrity a letter complainint of "cruel and discriminating treatment" at the hands of the school board. Yesterday, board vice chairman Martha V. Pennino of Centreville upbraided them for the comment.

"I did not appreciate what they had to say about the county. They think it's like the county is trying to do something inhumane, which just isn't the case," Pennino said. "To be so critical after having lived in Fairfax County only four years I find to be distasteful."