A Prince William County judge's order that a retarded woman be sterilized at her mother's request is being challenged by lawyers who question whether the Virginia law adquately protects the rights of the person being sterilized.

Judge Arthur W. Sinclair had ordered that the 20-year-old Dumfries woman receive an abortion and be sterilized. The abortion was performed last november after the woman became pregnant following intercourse with a juvenile during a family picnic. The woman has the intelligence of a 6- to 8-year-old child, and did not know she was pregnant, according to testimony in the case.

An abortion and sterilization were sought under a Virginia law permitting the action upon petition of a parent or guardian if the court determines "that the operation is in the best interest of such minor and society; and further that said minor is affliceted with any hereditary form of mental illness that is recurrent or with mental retardation . . . "

Virginia's statute is not unlike other such laws across the natin, with "some better, some similar and some worse," according to lawyers at the Mental Health Law Project, a nonprofit, public interest group seeking reform of laws affecting the mentally handicapped.

Judge Sinclair issued a court order last Oct. 21 for the operations after talking with the woman and conducting in his chambers a hearing with her mother, the family's lawyer and a court-appointed lawyer for the woman. The woman was not present at the hearing.

The mother testified that the woman was mentally retarded because of a childhood illness, and a teacher at the Woodbridge Activity School, which the woman attends, and a county school psychologist each testified that the woman was not able to care adequately for an infant.

There was additional testimony that the woman had never used nor received instruction in birth control techniques.

She was said to have made "tremendous advancements" in skills at the Woodbridge school.

Judge Sinclair said yesterday, "she was an extremely retarded girl. I had evidence from all over the place that bore out the legitimacy of the case."

Sinclair, who said it was the first sterilization case in his memory, said, "I guess any system could be abused," but that he had no doubt about the justification for his decision. The woman's abortion was performed in November, but Sinclair stayed the sterilization order pending appeal.

Raymond J. Morley, the woman's court-appointed lawyer, opposed the order and sought aid from the American Civil Liberties Union and the Mental Health Law Project.

Morley and project lawyers Robert Plotkin and Norman S. Rosenberg then filed a petition for appeal to the Virginia Supreme Court.

They argued that there is "no clear and convincing evidence" in the record that sterilization was in the best interest of the woman or society.

They contended that the law, like those in many other states, is "unconstitutionally vague and overbroad because it fails to provide adequate standards by which a court may reach a sterilization decision."

Morley said yesterday, "I talked to the respondent at great length. You would think that she was mentally retarded when you talked to her, but you can definitely have a conversation with her.

"I came to the conclusion that at this time she could not handle the responsibility of having a child, but that that was not necessarily so in the future."

Rosenberg argued that "the record must be sufficiently clear and exhaustive so that one can determine if it is in the best interest of the subject. With a matter so constitutionally fundamental, any decision must be supported by a thorough and adequate record.

"There was no evidence of trying birth control, no testimony by a doctor, no evidence that sterilization ws medically necessary," he said.

"Unless and until you have had independent medical and psychiatric evaluation and explored alternatives, it is impossible to know if this meets the needs of the child," Rosenberg said.

In the appeal Rosenberg compared the "vagueness" of Virginia's law to a North Carolina law that provides that a person to be sterilized must be found to "posses a physical, mental or nervous disease or deficiency which was not likely to improve," and either be "probably incapable for caring for a child" or be likely to bear a child that "probably would have serious physical, mental or nervous diseases or deficiencies."

He said the appeal would likely be taken to federal courts if it fails in Virginia.

The federal Health, Education and Welfare Department is now considering extensive revision of its rules with guidelines that would require a 30-day period between the signing of a consent form and the operation, and no sterilization of any person under 21.

HEW, in its proposed rules, said, "The department seeks to avoid any coerced sterilizations of the mentally incompetent.

" . . . There is reason to fear that any exception for sterilization of the profoundly retarded would create myriad possibilities for sterilization abuse . . . These issues are deeply troubling," HEW stated.