About 5,000 of the 23,000 District of Columbia voters who were forced to cast special challenged ballots in the city's Sept. 14 primary have been disqualified and told that their votes will not count, elections officials said yesterday.

Teddy Filosofos, who submitted his resignation earlier this week as executive director of the Board of Elections and Ethics, said yesterday that the 5,017 voters were disqualified because their registration cards were missing from the board's card file, or they failed to sign their challenged ballots, or they furnished insufficient information, such as a missing street name, on the special ballot envelope.

Although the board's action was not expected to affect the outcome of any primary results, the news of the large number of invalid ballots yesterday added to the outrage over the use of challenged ballots in the primary and past election foul-ups that have plagued city voters.

"Of course I was upset," said Annette Samuels, press secretary to Mayor Marion Barry and one of the voters notified by mail this week that the ballots would not be counted. "It the letter said I wasn't registered but I went down and told them I have my card."

Robert Kleinman and his wife, Deborah Agus, reregistered recently, after moving from Ward 2 to Ward 3, only to be told at the polls that they had not registered. They learned by letter this week that his vote was not counted, but hers was.

Both had voted in the District twice before and reregistered together at a McLean Gardens voter rally earlier this year. Kleinman said his wife had received a registration card in the mail but that he had had to vote a special ballot, which he now learns was not counted.

"All I can say is that I'm glad my candidate didn't lose by one vote," said Kleinman, who expects he will "probably" fill out the new registration forms he recently received in the mail.

Asked about the large number of disqualified voters,Filosofos said, I'm not concerned about the large number of special ballots that we had to disqualify. It's only one-fifth of the total number" of challenged ballots.

The names of disqualified voters living in seven of the city's eight wards will be printed in tomorrow's edition of The Washington Post. Ward 5 voters who were disqualified were listed in last Monday's paper.

Voters who disagree with the board's ruling were told in the letter that they may appeal by showing up at Room 409 of the District Building between 9 a.m. and 5:15 p.m. Tuesday.

Elections officials said yesterday that it was mathematically possible but highly unlikely that any disqualified voters who are later determined to be qualified, could affect the close races.

In the closest, which involved the council seat in Ward 5, William Spaulding defeated Robert Artisst by 713 votes, but there were only 434 disqualified voters in that ward. In other races, virtually all of the disqualified voters later found qualified would have to have voted for the same candidate to have an impact on the race, officials said yesterday.

Elections Board Chairman Albert J. Beveridge said yesterday he expects "several hundred" of the disqualified voters eventually to be found qualified and have their votes counted. Beveridge said that when 800 voters were disqualified in the 1981 school board election, about half were later determined to be qualified.

Filosofos submitted his resignation earlier this week in the wake of yet another election day foul-up, citing undue interference from city politicians in his efforts to straighten out the board's problems. Barry has said he will review the situation.

Elections officials have for years been grappling with ways to run an efficient election in the District, and have been hampered, according to Filosofos and others, by an inability to come up with a complete, current, accurate list of voters in the city.

There have been recent attempts to compile such a list using computers, but Filosofos, who came to the city earlier this year, termed much of the existing information "garbage."

On election day this year, an estimated 17 percent of the 124,000 D.C. residents who came to the polls were told their names were missing from voter rolls. Among those who were forced to cast challenged ballots were several candidates and city officials.

Filosofos said that election procedures improved this time because, unlike past elections in which people were turned away without being allowed to vote, there were enough challenge ballots at the polls to allow everyone who wanted to vote to cast a ballot.

Filosofos said he did not expect a mass rush on the District Building Tuesday, when disqualified voters have been told to appear to contest the board's ruling. He said when 434 voters were disqualified in Ward 5, only three showed up to contest the decision and another 73 telephoned the elections board.

He said voters listed as disqualified may call in with their registration numbers, in lieu of showing up, if they have valid cards. They then will be sent new cards, and their appeal will be processed, he said.

Filosofos said that disqualified voters were not told in the letter that they could make their appeal by telephone because by law, the appeal must be in person. If someone happens to phone in, he said, the board will bend the rule and take their information by phone.