VERGENNES, VT. -- There are few House races in the country where the challenger is better known than the incumbent. Rep. Peter Smith (R-Vt.) is painfully conscious that his is one.

At a candidates' debate sponsored by the Vermont Press Association here the other day, the congressman said, ''The gentleman to my right is a genuine celebrity -- a truly famous person. And there's a frustration (for me) in that.''

''The gentleman to my right'' is Bernard Sanders, the 48-year-old, former four-term mayor of Burlington, an avowed Socialist, whose uncombed shock of gray hair and strident Brooklyn-accented voice have become so famous in this state that his campaign buttons say simply: ''Bernie.''

On paper, Smith should be a cinch for reelection to Vermont's lone House seat, which Republicans have held for all but two years since it was created in 1932. A lanky, 44-year-old educator, with a B.A. degree from Princeton and two graduate degrees from Harvard, Smith is a former lieutenant-governor and GOP nominee for governor, very much in the moderate-liberal mainstream of Vermont Republicanism. As a freshman in the House, he has been an unusually visible and effective minority member of the Education and Labor Committee. The resolution he introduced calling for a special prosecutor on the savings and loan cases drew so much support in both parties that it helped prompt parallel action from the House Democratic leadership.

But Smith is indelibly tagged with the ''preppy'' label in the Vermont press; the word was used three times in a feature on politicians' clothes in Sunday's Burlington Free Press. And that is no help in a state with a growing number of counterculture immigrants, to say nothing of frugal Yankee natives.

But Smith's real problem is Sanders, a natural campaigner who covets the national prominence and influence he will gain if he wins what is certain to be one of the most publicized congressional races this year. Sanders migrated from Brooklyn to Vermont via the University of Chicago, part of what some Vermonters call ''the hippie invasion'' of the Vietnam War years. In the '70s, he worked as a carpenter and freelance writer and moviemaker. He also tried politics, but drew few votes as a senatorial and gubernatorial candidate of the radical Liberty Union party.

But when he upset the five-term incumbent by 10 votes to become mayor of the state's largest city in 1981, his political career was launched. He quickly cleaned out a nest of political cronies, brought in professional managers and tight accounting -- and focused his leftist rhetoric on the Reagan administration, while sparing local industries and developers.

Abandoning his Liberty Union label to run as an independent, Sanders finished third in the 1986 gubernatorial race with 14 percent of the vote, and second to Smith, with 37 percent of the vote, in the 1988 House contest. In that last race, Sanders got double the vote of the Democratic nominee, House majority leader Paul Poirier, and trailed Smith by only 8,000 votes.

This year, Smith faces a problem he did not have in 1988: a concerted attack from gun owners angry about his vote to restrict semiautomatic weapons. Gun shops are distributing bumper stickers saying, ''Smith and Wesson, Yes. Peter Smith, No.''

Meanwhile, Sanders has caught some good breaks. The likely Democratic nominee, University of Vermont professor Delores Sandoval, has taken more radical positions than his, advocating legalization of some drugs and opposing U.S. military intervention in the Persian Gulf. Despite Sanders' past role as a spoiler, many prominent Democrats, including gubernatorial candidate Peter Welch, have endorsed Sanders over Sandoval. Their support undercuts Smith's efforts to depict Sanders as a maverick who could only ''stand outside Congress and shake his fist'' if elected.

Sanders asserts that if he wins, he will be able to get committee assignments from the Democratic Caucus, while remaining a political independent. He has intrigued Vermont reporters with grandiose hints that his victory could trigger a wave of similar candidacies in 1992, reviving the energy of the Left in American politics and giving Vermont a claim to national attention.

Meantime, while vigorously denying that he is moderating his message, he is emphasizing only the most marketable of his old slogans -- a call for universal health insurance, for example, and a demand that taxes be raised on the rich to pay for the savings and loan bailout and to reduce the deficit.

On some issues, Sanders sounds downright mainstream. A strident critic of U.S. policy in Nicaragua and Panama, he now supports the embargo of Iraq and deployment of American troops to Saudi Arabia. Sanders says, ''We cannot turn our back'' on the threat from ''dictator-despot'' Saddam Hussein.

Speculating on the source of this new hawkishness, University of Vermont political scientist Garrison Nelson, a longtime Sanders-watcher, says, ''He's so close to winning this thing, he can taste it.''

And Smith can feel the threat from ''the gentleman on my right.''