If either Marion Barry or Sterling Tucker becomes the next mayor of Washington, numerous department chiefs in the city government can start cleaning out their desks. Their services will no longer be needed, the candidates say.

Barry is telling voters at numerous gatherings throughout the city that two-thirds of the 30 agency heads in the city government are "incompetent."

"I'm going to ask them to leave quietly," Barry says, his voice rising, "and if they don't I'm going to fire them."

Tucker says that if he is elected mayor he will ask for all the department directors to submit resignations. Then, he said, he will accept those from people he does not want to keep and tear up those of people he wants in his administration.

Barry's and Tucker's chief opponent in the Sept. 12 Democratic mayoral primary is Mayor Walter E. Washington. As the man who appointed the agency chiefs who his opponents are now attacking, he said it is "irresponsible" for Barry and Tucker to criticize the officials.

"One of the ways you can take morale to its lowest is to talk about sweeping everyone out," the mayor said in an interview. "I have no so-called hit list. It's unfortunate. It gets people upset."

The mayor said that he will examine the city government department heads "in light of winning the election" and may make some changes, but not until then.

While Barry, now a city council member, and Tucker, the current city council chairman, differ in approach to naming new department heads, they agree on one thing: Neither city housing chief Lorenzo W. Jacobs nor planning director Ben W. Gilbert will be part of their administrations.

They're the worst problems," Barry said. "Jacobs has no experience or background in housing. Mr. Gilbert is a former newspaperman and is not a planner or an administrator."

"A planner ought to head the planning office," Tucker said, adding that Jacobs "does not enjoy the public's confidence."

Jacobs, who on at least one occasion sat in an audience as Barry and Tucker announced their plans for firing him, is philosophical about the criticism. "They've said what they would do if a certain contingency occurs," he said. "I'm not concerned about that contingency happening."

Gilbert, a former editor at The Washington Post and a confidant of the mayor's, said, "I'm just going to do the best job I can day to day."

He declined comment on Barry's and Tucker's attacks on him, saying, "It's my personal reaction and I'm not ready to share it."

In addition to Jacobs and Gilbert, Tucker said that he would not retain city administrator Julian Dugas and acting city labor department director Thomas Wilkins.

Tucker said that he has made a "philosophical decision" that a person experienced in managing a municipal government should be named city administrator. Dugas is a lawyer.

When the mayor recently named Wilkins to his post, Tucker quickly issued a broadside on the appointment.

"Walter Washington is at it again," Tucker said. "Obviously the way to succeed in his administration is to fail. Tom Wilkins is the man who botched up our jobs program, who failed to use federal funds and planning programs that are available to him now - and is now in a position to botch up the whole program."

Through a spokesman, Wilkins declined comment.

Barry declined to name who else he plans to keep in his administration, but promised before the primary to release a list of Washington's appointees who "must go, a probably-go list and a maybe-go list."

He said he might keep on some of the agency chiefs in other jobs, except for Jacobs and Gilbert.

Tucker has pledged to "put an end to government by hacks, holdovers and cronies. In their place, we will install the finest minds and professional talent ever assembled in any city government."

"I may not keep as many as third" of the existing departments heads, Tucker said, adding that he wants departmental directors who will fit into his administration's "philosophical approach" (to running the city government), be able to "work as a team" and have the knowledge and ability to run specific agencies.

It is possible that Tucker and Barry could encounter some difficulty in forcing out a department head, if the official did not want to leave the D.C. government.

George Harrod, the city's personnel director, said current regulations would prohibit a mayor from summarily removing department heads - if they did not want to leave - without proving to the Civil Service Commission that they were incompetent. They could, however, be transferred to other jobs and given the same salary. Department heads generally are paid between $42,000 and $47,500 a year.

Under the city's proposed personnel rule changes, which would not take effect until Jan, 1, 1980, department heads would serve at the pleasure of a mayor, much like federal cabinet officials do for a president, Harrod said.