Charles E. Coe, the independent candidate for Congress who has not had it easy, paced around his home one day last wek in stocking feet and a white undershirt, grumbling about human rights. He has just suffered the cruelest cut of all.

His own alma mater, George Mason University in Fairfax County, had not invited him to a debate with his opponents, Rep. Herbert E. Harris II and Republican Fairfax Board Chairman John F. Herrity.

"That's not human rights," said Coe, a 39-year-old former insurance salesman, former mailman and somtimes substitute school teacher. "That's out-and-out indecency."

When Coe stopped pacing and sat down on a sofa next to a stack of newspapers with stories about the race in Northern Virginia's 8th Congressional District, he said he was going to let it bother him.

"I'm not going to stew about the debate for three or four years or anything, but it is upsetting."

The life of an independent candidate who has hardly any money any organization is chocked full of little upsets, Coe said.

There were the 1,000 "Chuck Coe for Congress" bumper stickers that cost him more than $100, but which nobody ("excepting those teen-agers who'll put anything on their cars") would use.

At the few debates he was invited to, there were the questions on issues that Coe says he doesn't understand. Asked about unsuccessful efforts by Congress to pass a college tutition tax credit, Coe said, in part, "Every young man and every young gal ought to go into the service to straighten their lives out."

There have, however, been happy moments in his 16-month run for Congress which has led him from front doors to grocery stores and finally back home in the last weeks of his campaign where he answers his phone. One of the things he did right, he says, was buy sponges.

The sponges - they are dehydrated, highly compressed sponges - costs Coe $120. They are printed in blue ink, "Make Chuck Coe Your Next Congressman." In small type the sponges says, "Dip In Water - See What Happens."

"I kid the women when I pass them out door-to-door. I say my name will never fade from that sponge," Coe said.

Coe, who lives with his wife and two children in the Mount Vernon section of Fairfax and who moved to Northern Virginia when he was 4 months old, said he decided to run for Congress when he heard that Congress, in January 1977, voted itself a pay raise.

"I figured what the hell is going on?" Coe said.

"It was really doing my heavy campaigning that spring (of 1977). If you know you need money, and I knew I needed money, you got to knock on doors. When I got inside, people would usually know that I needed a few bucks," Coe said.

Using that method of fund-raising, Coe has collected about $2,000. His Republican opponent, Herrity, has raised more than $126,000 and his Democratic opponent, Harris, raised more than $100,000.

Coe said big money has no part in the American electoral system. "I would never be part of charging people $50 to come have dinner with me. "I'm not a king. I'm an average guy. Oh. I could see charging maybe $10 or perhaps $25."

In the joint television appearances Coe has been invited to because of equal time provisions in federal broadcasting law, the independent candidate says he sometimes sounds comfused. Coe recognizes the problem and said he will have a better understanding of the issues facing voters in the 8th District after he's elected.

"I know what I'm talking about," Coe explains. "But I like to be simple. Sometimes I fail and I have a tough time even following myself. I think a lot of use have that problem."

Coe accuses his opponents of not knowing he exists, but say he does not care.

"I think I have a good chance to win as far as my own perceptions. What the public perceives is their own business."