James C. Sanders, head of the Small Business Administration, accused budget director David A. Stockman yesterday of "slander" for calling his agency "a billion-dollar waste, a rathole."

Stockman, who is spearheading the Reagan administration's drive to abolish the agency, made the comments at a breakfast meeting with reporters Wednesday. And Sanders, after reading the remarks in The Baltimore Sun, shed his usually mild-mannered demeanor in a brief interview.

"It's unthinkable that anyone in a responsible position would use that kind of excessive and abrasive language," Sanders said. "I just think it's a despicable term.

"You can't call an agency a rathole. That slanders the achievements of everyone in the agency . . . . I'm sure he'll either retract that statement or qualify it."

Stockman's office said he would have no further comment. But the war of words is likely to resume next week when the Office of Management and Budget director testifies before the Senate Small Business Committee. Both Republicans and Democrats on the panel have said they do not want to kill the SBA.

Stockman and Sanders have been at odds since the OMB proposed to eliminate the small business agency without telling Sanders in advance. While also criticizing farm subsidies and military pensions, Stockman has appeared to take special joy in denigrating Sanders' agency.

Sanders has made it clear that he wants the agency to survive, although he wants to drop some of its most expensive programs. But he has gone out of his way to be a team player by not challenging the administration's proposal in public.

Yesterday, however, he said Stockman's rhetoric "creates hostility" and makes it more difficult to win approval of the president's budget.