The security chief at Great Oaks Center in Silver Spring, Maryland's second largest institution for the mentally retarded, has resigned, saying that he could not work with the facility's new director who criticized his investigation into a recent assault there.

William T. Avery, who was in charge of four guards, said he quit his job after Director Marvin Malcotti told him he was "not satisfied with the investigation" of an assault on a severely retarded 26-year-old mute woman last week.

"My department was investigating the incident when he said he was not satisfied with what we were doing," said Avery, 47, a former District police officer who had worked at Great Oaks for four years. "I cannot work with someone like that. He's a little dictator."

Malcotti, director since January, said yesterday he would not discuss Avery's resignation or the investigation. Asked if Avery's resignation was related to the assault, Malcotti said: "It's certainly not, in my view, a causative relationship. I would have to say it's somewhere in the area of coincidence."

Avery's resignation comes as Prince George's County police continue to investigate the attack on the resident July 14 and an attack on a 38-year-old retarded woman there three days later.

Center employe Patrick Joseph Obryhim of Elkridge, Md., was charged with second-degree rape in the second incident. Police said yesterday that the victim, who has the mental abilities of a 6-year-old, had been interviewed about the attack.

No charges have been filed in the July 14 incident, which occurred after the 28-year-old woman apparently strayed from the center, police said. Evidence suggests that the woman was raped, but she cannot say what happened because of her disability, police said.

Great Oaks, a facility for about 420 mentally retarded children and adults, has been the focus of scrutiny by the state in the last year.

In 1985, the state's Mental Retardation and Developmental Disabilities Administration investigated the facility at the request of an employe union, and issued a report saying Great Oaks was experiencing a "significant long-term employe management problem."

As a result, Director Clifford Lockyer was reassigned as the coordinator for community programs in several counties in Southern Maryland. Deputy Director Paul Joyce was reassigned as assistant superintendent at Rosewood Center in Owings Mills near Baltimore, the state's oldest and largest institution for the mentally retarded.

Malcotti, 46, succeeded Acting Director Edward Orndorff, who had been appointed to the job after the administrative shake-up.