The President's Committee on Mental Retardation voted unanimously yesterday to take a stand against that of the Justice Department in a case before the Supreme Court involving the rights of mentally retarded people to live in group homes in residential areas.

In a separate issue, the committee's executive director criticized portions of a settlement announced Thursday between the Justice Department and Rosewood Center near Baltimore, which set minimum standards for institutions for the mentally retarded.

The 21 committee members, 15 of whom were present yesterday, were appointed by President Reagan. Eleven of the 21 have sons or daughters who are mentally retarded, said committee member Larry Kane.

The resolution that the committee voted to send to the Supreme Court states that a zoning ordinance in Cleburne, Tex., prohibiting group homes for the mentally retarded in residential areas "represents a type of unlawful, invidious discrimination existing across the nation which is barred by the U.S. Constitution."

The 5th U.S. Circuit Court of Appeals took the same position, ruling the ordinance unconstitutional. But an amicus brief filed by the Justice Department last month challenges that ruling.

"We were torn," said Vince Gray, executive director of the committee. "We're advocates for the mentally retarded but we also want to support the administration's views . . . . We ended up speaking to the unconstitutional nature of the ordinance" in supporting the court's decision.

Cleburne's city zoning ordinance requires a special-use permit for the operation of any "hospitals for the insane or feeble-minded, or alcoholic or drug addicts, or penal or correctional institutions." The city council denied such a permit for a residential-area group home for the mentally retarded.

The Justice Department's brief does not state whether it believes it is unconstitutional to prohibit such group homes in residential areas. Instead, the brief challenges the appeals court's legal reasoning, saying that the panel applied an overly strict standard in determining that the city ordinance violates the equal-protection clause of the 14th Amendment.

Committee members said the standard recommended by the Justice Department is too lenient and would result in a greater likelihood that mentally retarded persons would not be allowed to live in group homes in residential areas.

"We feel that this case has the potential of being a Brown versus Board of Education for the mentally retarded," Kane said, referring to the Supreme Court's landmark 1954 decision on school desegregation. "If the high court reverses the decision, it will ignore a century and a half of discrimination."

Assistant Attorney General William Bradford Reynolds denied that the department's brief was contrary to the committee's resolution in support of the appeals court's ruling. Told that the brief supports reversal of the ruling, Reynolds said: "You ought not to strain so hard to find a conflict. Our reversal motion is only aimed at the standard applied. We don't urge the Supreme Court to validate the ordinance.

"We even say in the brief that we have serious reservations about whether that ordinance would survive the test that should be applied. We get certain issues before the court. We can't reduce them to whatever result might be down the road."

During an interview, Gray also criticized portions of the Justice Department's recent settlement involving conditions at the Rosewood Center, an institution for the mentally retarded, saying the settlement has a "very strange-looking definition of training" for residents.

Gray noted that the settlement only provides training to help the mentally retarded stay free from harm, but does not provide for training to help residents become independent and functioning members of society. However, he said that the staffing levels in the settlement appeared adequate, with the possible exception of the ratio of one psychologist per 20 residents.