Charles Town Race Track was spared some potentially devastating competition when Virginia's electroate rejected a proposal to legalize pari-mutuel wagering. That was the first bright news for the track in a year of travail, which has seen its business decline and the quality of its racing deteriorate.

But the verdict of Virginia's 1.1 million voters on Tuesday was probably less important to Charles Town's well-being than a verdict rendered by about two dozen gamblers.

Harry typified them. He used to travel six nights a week from his Silver Springs home to West Virginia, lured by the gimmick bets - the twin double, the trifecta, the jackpot - which have long been Charles Town's main attractions. Harry had found a way to earn his livelihood from them.

He approached the gimmick with a bold betting strategy rather than finesse. He would invest hundreds of dollars in a race, using as many as seven horses in a 10-horse field, eliminating weak favorites and emphasizing plausible longshots, trying to hit the gimmicks when they produced blockbuster payoffs. His method worked. Once he collected three payoffs of more than $15,000 in a single week.

But the big winner was the race-track. Harry probably pushed $10,000 a week through the betting windows at Charles Town. Along with a small coterie or other gimmick-bet specialists, he was a mainstay of the track's business - until the disastrous day of May 18, 1977.

The Internal Revenue Service had long been fretting that it did not collect its fair share of big payoffs at the track, and so on that date it instituted a 20 percent withholding tax on payoffs that returned more than $1,000 for $2. Instead of collecting a $15,000 payoff, Harty would now collect $12,000 and have to fight the IR on the next April 15 for the money it had with-held. At about the same time, West Virginia increased the state "take" on gimmick wagers to an exorbitant 25 per cent. Now the original $15,000 payoff was shaved to about $11,000.

Harry saw that betting the gimmicks had become a hopeless proposition, and he quit Charles Town for good. Now he does most of his gambling on gin rummy.

His reaction was typical. "Some guys pulled out immediately," said Bill McDonald, Charles Town general manager. "Others tried to stick it out for a while, but finally left. I can name at least 20 guys who used to put no less than $2,000 apiece into the pools every night who aren't here any more."

Not only has the withholding tax driven away Charles Town's biggest gamblers, but it has eaten up the capital of the players who have stayed in action. Racetracks depend upon the recirculation of the betting dollar for their business, but the IRS grabs the dollar and removes it from the track. Since it instituted the withholding tax it has confiscated more than $1.5 million of Charles Town customers' money. Rarely do they ever see it again.

Hurt by the witholding tax and such other factors as the extension of the Maryland harness-racing season, Charles Town's attendance and handle have fallen. The track does not have enough money to offer purses competitive with other racing states. A $3,000 claiming horse at Penn National Race Course Town, it would run for a $2,300 purse.

As a result, the owners of legitimate $3,000 animals have no incentive to run them at Charles Town, and so the track's racing programs have become increasingly populated by rock-bottom animals too cheap to race anywhere else. The poor quality of the competition presumably hurts the track's business even more.

All this is a pity, because Charles Town is one of the most pleasant tracks in the area, with a management responsive to the desires of its customers. That management has not given up. McDonald said Charles Town has doubled its advertising and promotion, attempting to lure new fans. The track would like to institute Sunday racing. It hopes to persuade the state to reduce its takeout on gimmick bets.

With the threat of competition from Virginia laid to rest, Charles Town will surely survive, and with Sunday racing it might even prosper again. But it will never flourish as it used to until the Internal Revenue Service relents, and permits Harry and the other heavy hitters to get back into action.