Virginia's state personnel director yesterday called on the state government to abolish affirmative action plans that give special employment advantages to minorities and women, but withdrew his recommendation hours later under a firestorm of criticism.

"Very simply and honestly, affirmative action and equal opportunity are incompatible," said Kenneth B. Yancey, who oversees the state's efforts to recruit women and minority workers. "Affirmative action requires discrimination in the workplace rather than equal employment opportunity." He called affirmative action "a crutch that is no longer needed."

Within hours of Yancey's early morning announcement, Republican Gov. John N. Dalton responded to protests from state equal rights groups and both gubernatorial candidates by repudiating the man he had appointed to the personnel post three years ago. "Mr. Yancey has not spoken to me about any dilution of Virginia's affirmative action program, nor have I authorized any change in my executive order on affirmative action ," Dalton said in a statement released by his office.

"That executive order remains in full force and effect, and I have no intention of making changes to it," Dalton said.

Yesterday's incident was the latest controversy in what has been a stormy tenure at the top of the state's personnel system for Yancey, a man who has a reputation as an irascible, shoot-from-the-hip administrator. He was the subject last year of a job discrimination suit filed by one of his employes, a Japanese-American who contended she was denied a top post in the agency because Yancey wanted to hire a less qualified black. That suit was settled out of court for $17,500.

Yancey has also been criticized by Dalton's own Equal Employment Opportunity Committee and members of his department for allegedly lax enforcement of state and federal job bias rules, and has been accused of cronyism and political favoritism in departmental hirings.

In the announcement he released yesterday morning, Yancey maintained that the state has made strides in ensuring equal employment for women and minorities over the last few years, thus making the state's recruitment campaign for the two groups unnecessary. He said minorities hold 25 percent of all state jobs, far in excess of their 16 percent share of the state's labor market, and that women, who account for 38 percent of the state's work force, make up 52 percent of the state's employes.

But equal rights groups say that Yancey has done little to reverse the Virginia state government's traditional hiring preferences -- white males at the top, and minorities and women at the bottom.

"He might tell it that way, but the NAACP is not going to allow this lie to live," said Jack Gravely, executive secretary of the Virginia branch of the NAACP. Gravely called Yancey's statements "outrageous, unbelievable and simply not true.

"Black folks still polish the brass and cut the grass for the state of Virginia and we're sick and tired of it," said Gravely. "And women still type the letters and make the coffee."

Salary statistics gathered by the federal Equal Employment Opportunity Commission seem to support the argument that women and minorities are clustered near the bottom of Virginia government pay scales. According to the commission's most recent figures, blacks who work for state and local governments in Virginia averaged $8,992 in 1979, 28.4 percent lower than the average pay of $12,554 for white males.

The same figures showed women making an average of $9,614, or 23.5 percent lower.

Judy Goldberg, Virginia spokeswoman for the American Civil Liberties Union, called Yancey's proposal "baseless and astounding," and urged him to step down. "Any official who disagrees with one of the major policies of his agency should either retract the disagreement or resign," she said.

Yancey said his recommendations were aimed not only at Dalton, who leaves office in January, but also at the two candidates who are vying for the governorship in the Nov. 3 election. But by midafternoon, both those candidates, Democratic Lt. Gov. Charles S. Robb and Republican Attorney General J. Marshall Coleman, had joined Dalton in rejecting the idea.

"We think the governor's executive order No. 1 calling for affirmative action should remain as a tenet of statewide policy, and it will be honored fully during the Coleman administration," said Coleman press secretary David Blee.

Robb, in a statement issued from the national lieutenant governors' conference in Philadelphia, said: "I remain committed to affirmative action and my administration will work faithfully to see that additional progress is made with equal employment at all levels of state government."

By the end of the day, Yancey seemed to have given up on the proposal for the time being, although he did not rule out a similar attempt in the future. "The governor has made it pretty clear that he's not interested in these changes, so these recommendations won't be made to him within this administration," Yancey said.