Less than 10 miles from here, across the West Virginia border, a storm is brewing over the issue of gambling, as the opponents of horse racing urge Virginians to reject pari-mutuel betting in a November referendum.

But here in the West Virginia panhandle, where horse racing has been a reality for 45 years, there is little question about its merits.

Charles Town, as a policeman joked to a visitor the other day, is a "one-horse town, or more accurately, a two-track town. Horse racing is its lifeblood. And many of the citizens of surrounding Jefferson County are worried that Vriginians are going to take their horses away.

"This town wouldn't go without racing," said Junior Ennis, proprietor of the Jefferson Newsstand, where much of the local gossip is traded among stacks of periodicals and a dozen brands of chewing tobacco.

A look at Charles Town offers an insight into racing's possible impact on Virginia, for in Charles Town, racing means money.

"If the races are not running, it's like a factory being closed down," Oliver Kastle, president of the local retailers' association, said. "It brings in a tremendous amount of revenue to the immediate area."

Like many small towns in America, Charles Town has come to be dominated by a single enterprise, and any threats to it makes people uncomfortable.

"I can assure you," said Kastle, that any time somebody mentions the races are going to shut down, everybody gets a damned bad look in his face."

If Virginians approve the referendum on pari-mutuel betting, race tracks are likely to be built in Norfolk and in Northern Virginia. Track officials here are punlicly less pessimistic about the possibility than we the townsfolk.

We won't see the race track go down the drain, but it would be nice not to have the competition," said [WORD ILLEGIBLE] R. Sussman, vice president of the Shenandoah Corp., which owns Shenandoah Downs and the Chales Town Turf Club.

The money tracks handle has been declining nationally in recent years, he noted, and local businessmen say that even if the track ekes by, many of them will not. They point to the business the tracks bring and the local payroll they generate.

The Shenandoah Corp., bought last year by the Keaton Corp., of New York, owns both tracks just outside town. Shenandoah, where the first half of this year's meets were run, is closed while the Turf Club handles the second half. Together, they account for 270 nights of racing annually.

The track combine, directly and indirectly, employs 5,500 workers on the premises, keeps dozens of vendors in business, brings about a million visitors to town each year, and keeps a string of motels, stores and restaurants alive.

The daily payroll exceeds $10,000, and the daily payout to horse owners is about $24,000, of which 90 percent remains in the local community to cover food and expenses for horses and attendants, according to the track general manager William McDonald.

Perhaps racing's impact is most notable during the off-season, when Mondays are cut from the summer schedule that otherwise runs everyday but Sunday. Locals call it "dark Monday" and close their shops early.

Last winter, when racing was knocked out for 15 days by snow and four more days by power shortages, the area felt the impact, Local retailers say. "It was like a ghose town," said Louis Luccarelli, who runs the local pizza restaurant.

In 1977, when the January opening was to be delayed two months at the Turf Club, local residents stirred a storm of protests, forcing the track to open in February.

Having two race tracks, residents point out, does pose problems. "The problem is the traffic," said Junior Ennis.

"For 3,000 people, you name me any town in the country that's got eight policemen," said Mayor Donald C. Naster. "But you name me any town with 3,000 people that's got two tracks."

Master also points on that the town gets no real estate taxes from the race tracks, which are beyond the city limits.

An annexation attempt in the late 1950 failed, and Master said another is unlikely. Despite the tax base racing might provide, "We don't want a situation where the race track ends up buying itself a mayor and a city council," he said.

Racing's political influence apparently is limited in both the town and the county. Only one of the eight city council members has a track job, for instance, and it is only parttime.

Local racing provides more than $6 million a year to West Virginia's state treasury, some of which reverts to Jefferson County as school aid. The county also gets real estate taxes on the property, and since Charles Town is the county seat, it does not completely suffer.

The spectators who come to the Charles Town races seem to mix well in town.

For the most part, they're people who are out for the fun," said John Newcomer, a partner in the Cliffside Motor Inn four miles outside town.

"It's all classes and income groups, a lot of busloads of people who might come twice a year," according to Richard Sanderson of Herndon, Va., a taxi driver who often brings a cabful of fans out three times a week from the District of Columbia.

A track survey shows that nearly 70 percent of the spectators drive more than an hour to get to the races, mostly from Maryland and the metropolitan District areas.

Crime has not been a problem. Neither the state racing commission northe U.S. attorney's office in Wheeling, which covers the area, has any record of race fixing or scandal.

"We run a tight ship," said Eileen Ledford, executive secretary of the commission.

One local businessmen emphasized that racing is not Charles Town's only business. Nearby plants of 3M, and Dixie Narco Co., a brass works and paperboard will employe several hundred people.

"There are a good many people who are antiracing," he said, "and they feel that if the track weren't here, something just as good would come along."

But racing opponents are difficult to find. Local industries employ fewer people than the track does horses - it boards 1,500 - and most local businessmen resent antiracing talk.

"I think it is a major economic base for the community, whether people like it or not, and I think it should be recognized as such," banker John Christmas said.

Most residents do, in fact, recognize the heavy role racing plays, and seem not to mind. For despite the traffic, the fanfare of racing and the thousands of visitors who come through daily, Charles Town has preserved a small-town atmosphere that exudes its history.

Its downtown is essentially the intersection of two streets, one called George, the other called Washington, and both named after the same U.S. president.

Washington, who laid out the town, named it after his brother, Charles, and their descendants still live here. It is said that the local cemetery contains more Washington than any other in the country.

"They may have slept somewhere else, but they're buried here," said the mayor.

The post office on one corner of the main intersection boasts that Charles Town was the site of America's first rural free delivery in 1896. On a second corner is the court house where John Brown, the abolitionist, was sentenced to be hanged in 1859.

Today, the only thing resembling a local controversy is whether racing will be extended to Sundays by the state legislature.

Some local clergymen oppose it, but they are heavily outnumbered. The more pressing question is what will happen in Virginia.

"If they ever go racing next door it's going to hurt the hell out of it," says a local restauranteur.