"There's a time to quit" is the enduring adage in boxing. Only a few fighters know when to walk out before they get carried out or zonked out. For the defenders of boxing, quitting is also the question. The time is now.

Like cornermen not realizing their staggering man is about to go down for the count, boxing boosters are now confronted with the American Medical Association. The doctors, facing facts long available, have called for the abolition of both professional and amateur boxing.

In calling for a ban on what sportswriter Jimmy Cannon saw as "the ugliest of all sports," the AMA correctly focused the issue as one of health, safety and deliberate violence. Dr. Joseph Boyle, the AMA president, said, "It seems to us an extraordinarily incongruous thing that we have a sport in which two people are literally paid to get into a ring and try to beat one another to death, or at least beat [one another] into a state of senselessness, which will leave them permanently brain-damaged."

Instead of counterpunching to that fact, the defenders of boxing become bobbers and weavers who swing back wildly with arguments that have nothing to do with the deadliness of boxing. They offer three:

* Boxing is the route to Easy Street. Teaching ghetto kids how to brain-batter each other is presented as a way out of the ghetto. Look at Sugar Ray Leonard, Marvelous Marvin Hagler and the others who fought up from poverty. Where do these rich doctors get off denying opportunity to the poor? If you let the defenders of boxing talk on, they would present prizefighting as superior to the Job Corps, Head Start and the Job Training Partnership Act put together.

Despite rare exceptions such as a Leonard and Hagler, the AMA believes that brain damage and other neurological disorders are common among the nation's estimated 10,000 professional boxers. Riches? They have gone from rags to stupor.

The escape-from-poverty argument begs another question. What about the young women of the ghetto? Perhaps they should seek the welterweight title in mud wrestling and all its millions.

* Ban coal mining, not boxing. Because other trades are more injurious than boxing, it is said, why don't the AMA and other tender hearts go after them? Two reasons exist for singling out boxing. It is a sport with lethality built into it. The object is to injure or kill. The fight crowd dances around that fact by saying, no, the object is to win. That's akin to arguing that the object of sky-diving is not the art of the fall but defeating the earth should you hit it chute unopened.

Boxing is not like coal mining -- or football, hockey or other dangerous professions -- because the fighters receive few of the normal job protections. Coal miners can get black-lung benefits. As yet, battered-brain benefits are not available to boxers.

Last year in hearings on the Federal Boxing Protection Act before the House subcommittee on labor standards, Rep. Pat Williams (D-Mont.) spoke about the absence of pension plans, insurance, profit sharing and unions for the endangered boxer: "This country needs to protect one of the few remaining workers who are unprotected . . . [He] is involved in one of America's most perilous occupations and yet remains virtually unprotected with regard to any legal sanctions."

* The AMA should mind its own business. This is a pet argument of Bert Sugar, the former publisher and editor of Ring magazine and a wearer of the championship belt for lyricizing about "the heroes, thrills and memorable moments" of boxing. When I discussed the AMA decision with Sugar, he grumbled that the doctors are elitists, the AMA represents only half of the nation's physicians and the organization was socially regressive for its past opposition to some health programs. What all that had to do with health, safety and violence, the bitter Sugar didn't say. He wasn't overly game, either, for a discussion of the impoverished families, widows and orphans and others who have mourned at hundreds of requiems for heavyweights and featherweights.

Sugar, companionable and quick- witted when you get him off the subject of human beings bashing each other, testified last year before Congress on the boxing protection act. He called for a reform of the sport as it is overlorded by the self-serving World Boxing Association and World Boxing Council. Sugar wants help from Congress to reform the sleazy, exploitive and violent fight racket.

The AMA solution is sounder: Ban it. Trying to reform boxing is like trying to clean a toxic waste dump with a drop of Listerine.