Fred Silverman, the only man in television history to have held key programming positions at all three networks, said yesterday he would "never ever" go back to ABC as head of programming, nor to any other network in a similar post. "You'll see me back at ABC when an icicle survives in hell," said Silverman from Los Angeles, where he is now an independent producer.

The former president of ABC Entertainment and former president and chief executive officer of NBC, Silverman, who began his network career as a CBS executive, was responding to rumors now circulating in Hollywood that ABC had entreated him to return to its upper echelons so as to resurrect the network from its current third-place position in prime-time ratings.

Known in his heyday at ABC as "the man with the golden gut" for his highly commercial programming and promotional instincts, Silverman left ABC in 1978 to accept a post with a loftier title at NBC. But his three years there saw that network's ratings worsen and its profits fall to record lows. He remains a television legend nevertheless, one who gave TV "Roots" and "Hill Street Blues" in his best moments and "Me and the Chimp" and "Supertrain" in his worst.

A high-ranking ABC spokesman suggested yesterday that rumors about ABC sending a distress signal to Fred Silverman were preposterous and had no basis in fact.

However, the idea that ABC is in deep trouble has a firm basis in fact.

In the November Nielsen "sweeps" just concluded, CBS was in first place for the month with a 16.8 rating (percentage of TV households) for an average prime-time minute and a 27 share (percentage of TV homes with sets in use). NBC was second with a 16 rating and 25 share. ABC was third with 15.2/24. That means NBC is up 5 percent from its showing in the November sweeps a year ago, CBS is down 12 percent and ABC is down 14 percent.

This was, according to sources at another network, ABC's worst November sweeps since 1968 and the first time since 1974 that the network has finished third. November is one of three sweep periods during the year when intensive ratings are taken that help determine what the network and its affiliates will charge advertisers.

Asked if he had been approached by anyone from ABC, Silverman said he'd had "a couple of conversations with people who are close to the situation" and that "it's a nice rumor," but stressed that he would not consider any offer to return. "I would never, never go back to a network in that kind of job," Silverman said. "Never.

"I'm 47 years old; I don't have that kind of energy any more, and I don't need that kind of grief," Silverman said. "ABC is in very bad shape. The problems that ABC is having now are as large as the problems we had at NBC. And it's not just prime time." Silverman was asked where ABC went awry. "I don't know," he said. "It didn't happen overnight. The problems are not unsolvable, but it could take a long, long time."

Silverman said he has just signed a contract with Walt Disney Studios to produce theatrical films that will be released under the Disney banner; the deal will be officially announced Monday. He also said he was pleased for Brandon Tartikoff, the boyish president of NBC Entertainment, which is enjoying an unusual season of vastly improved ratings. It was Silverman who installed Tartikoff in the job he has now, before resigning in 1981 and being succeeded by current NBC chairman Grant A. Tinker.

Silverman did not say he was enjoying ABC's misfortune. When he left ABC, there were bitter feelings on both sides; Silverman felt he was under-rewarded for his performance, and ABC Inc. president Frederick S. Pierce felt Silverman betrayed him by going to another network after allegedly promising not to. But Silverman did say of ABC's troubles, "It's nice to be on the sidelines. For once, it isn't my grief."