IT'S ALL over with me," says Phil Silvers, 66. "I'm not working as I should. The phone has stopped ringing."

There was a time, says Silvers, "when I was the top celebrity in the World excluding Bob Hope." In the '50s, he recall, his weekly "The Phil Silvers Show" put Milton Berle off the air."

Silvers is reduced to now-and-then guest appearances on shows like "Charlie's Angels" and every-once-in-a-while two-scene roles in off-the-lot films like the upcoming "Racquet," which is scheduled to open next month.

"'Racquet' was a one-day shoot. I did it for no money. I did it for what I used to tip the doorman. But I've never been funnier in four minutes."

Once know around the world as the tough-but-nice Sgt. Bilko, Silvers today works for " $13 a job." He even has trouble getting an agent. "It's a Mork and Mindy' world," Silvers sighs.

What happened?

Silvers has been ill. "I recovered, but nobody knows it," has says. Silvers desperately wants to be a star again.

Seven year ago, Silvers suffered what he calls "a slight stroke" when he was at the height of an earlier comeback.

"It was during the Broadway run of 'A Funny Thing Happened on the Way to the Forum,'" Silvers recalls. He can't resist interrupting himself: "Did you ever hear of anyone winning a Tony for a revival?"

Yes, Silvers won the second of his two Tonys for a revival?"

Yes, Silvers won the second of his two Tony for "Forum." He'd like no one to forget that.

There I was applause, applause, and one morning I woke up and I couldn't reach the telephone.

"I'm fine now. See the fingers?. See the arms? But it took a long time to recover becaused I gave up. I wanted to die. Comedians are all tragic figures. I wanted to die. All I could think of was, I closed the show, I closed the show.

"Then I went into a depression. I'm out of it now. I'm fine, but nobody believes me.

"Do I look like I'm ill? I'm well. If anything, I'm too well. Do you think I come on too strong?

Silvers does indeed come on strong. He worked hard for everything he achieved, and he can't understand why hard work won't get him what he wants now.

Wolfing down a Brown Derby lunch, carrying on the threads of a dozen stories at once, Silvers comes on like a dynamo without a flywheel. He still has the power, but there's no work to do.

"What you're seeing now is . . . it's dreadful to be a character, and to be aware of it. The reason I talk so much is that I've been with the best. I've been around so much. . . ."

At the age of 10, Silvers was singing professionally in New York. This was just after the First World War.

He worked his way up through the ranks of vaudeville to the top-the Minsky circuit. In the movies, he began in two-reel comedies and graduated during the Second World War to features.

Fox used him in 23 wartime comedies, playing the hero's pal in pictures like "My Gal Sal" and "You're in the Army Now." Today, Silvers lives in a high-rise near the Fox Studios.

"This building leans out over the place where I spent all those years following Victor Mature around. I always had the same line in every picture. 'I got the stuff in the car.' I never did find out what the 'stuff' was."

Silvers was successful enough on Broadway after the war and in nightclubs, but his snappy delivery and heavy black-rimmed glasses didn't become familiar all over the world until he went on TV in 1955 in a program called "You'll Never Get Rich."

Silvers' Sgt. Bilko character stole the show, which was quickly renamed "The Phil Silvers Show." "Mr Eisenhower had it every Wednesday morning in the White House," Silvers recalls.

"One day I blinked and the show closed. CBS claimed it was too expensive. They had to pay a whole platoon of actors.

"They canceled us at our height, despite mammoth ratings, because they wanted to get their money back in syndication. They had 148 episodes, and pretty soon they wished they had more.

"It had its 50th rerun on Channel 13 not too long ago and it outrated the network programs. In Australia, still, I'm an idol. I can't cross the street in England. I don't get any money for it now, but I did for years."

Even if Silvers hadn't been up there with Berle, Ball, Webb, Caesar and Sullivan, he'd be remenbered for his postwar Broadway long-runners "High Button Shoes" and "Do Re Mi."

Today, Silvers feels he's not sufficiently remembered. He mentions being taken to the Brown Derby a year or two ago by one of the top agents in town.

"What a romance this guy handed me. I knew it was a brush-off.He pushed every button on his desk, and his whole staff trotted in, the oldest one of them 11 years old. He told me what they were gonna do for me.

"I never heard from him again.

"No names, I'll tell you something else. I did a test for the TV 'Sunshine Boys.' The Day I was there, the hall was full of incredibly talented old actors who had never tested before.

"My test was brillian. By nor, I got no time left for nonsense. I don't say I'm good when i'm not.

"I never heard anything, so I called the director. He said he and the producer had agreed. It was no contest, I was so brilliant. But the producer turned me down. 'Silvers has been ill. I'm afraid he won't stand up.'

"So I called this producer. I told him, 'I'm a compulsive gambler. I got five kids to support. I got a doctor's certificate. I'm well. You're ruining my career.'

"The show lasted one night. With Red Buttons."

Silvers has had similar disappointments. Once a part he was up for went to - "You won't believe it" - Eva Gabor.

So Silvers survives doing guest shots, awaiting the break that will put him on top again, though he fears it will never come.

"You do something small, as a favor for a friend, and it's like a fingerprint.

They never let you get over it." He mentions one possible upcoming film role this summer, "as O.J. Simpson's manager."

Two ideas keep him going-dignity and revenge.

"Even in burlesque I always had dignity," Silvers claims. What is dignity in a comedian?

"Dignity is never selling yourself short. I never said a single ribald line in pictures, not until 'Racquet.' Dignity is not saying four-letter words-but when you say'em, say'em good.

"Dignity is not telling racial jokes," Silvers adds. Almost immediately, this fascinating, contradiction-ridden man launches into one of the most ribald racial jokes ever told at the Brown Derby.

As for revenge, Silvers has many enemies. They don't know they're enemies. They don't know he exists. That's why they're enemies.

They include his former agent ("I was his first client. Now he's gone into production and he's getting his asshanded to him") and a famous librettist ("His show was a flop on the road till I changed every line, and it ran three years on Broadway").

Above all, Silvers hated being made to feel small, even when it's unintentioanl. He recently appeared on the "Merv Griffin Show" and was challenged to prove he could play the clarinet.

"I made Merv Griffin a star when I turned him down for 'Do Re Mi.' He'll do anything I aks. I gotta go back on his show and get even with him.

"On the air he gave me a clarinet from a guy in the orchestra, and said to play it. I have as good a tone as any clarinetist, but I have as good a tone as any clarinetist, but I don't read music. I wasn't prepared."

To viewers, Silvers appeared to play very well, but Silvers wasn't satified. "I'm going to go back on, for revenge. I'll go back with dignity. Get me my corporals. I'll get even with those guys.

Thoughts of revenge keep a lot of people alive. Silvers is one of those people who has forgotten nothing and learned nothing.

"When you know pain, you change," he says. "I'm in constant pain. Not from the stroke. I licked that.

"They took away the track. I'd be an animal if I ever went to the races again. I'm finished with the track.

"I'm finished with the business, too. But they'll never take my talent away." CAPTION: Picture, no caption