Democratic D.C. delegate nominee Eleanor Holmes Norton went to the heart of Ward 3 yesterday to start rebuilding her support among white voters, support she said was badly damaged by the disclosure that she and her husband had not filed District income tax returns since 1982.

Shaking hands at the Cleveland Park Metro station during the evening rush hour, Norton thanked people for helping her win Tuesday's primary election, in which she received 40 percent of the vote in a five-candidate field.

Only two of the more than 100 people whom Norton greeted refused to shake her hand. One man dressed in a conservative business suit walked by smiling, shook his finger in her face and said, "You should pay your taxes."

Norton didn't respond except to smile and take another sip from a bottle of lemonade.

Norton has said that she was unaware that her D.C. tax returns for 1983 through 1989 were not filed because her husband handled all of the family's tax matters. Failure to file a tax return is a misdemeanor, but no charges have been filed against Norton or her husband.

Most of the people she encountered at the Metro stop offered congratulations.

One woman appeared surprised that Norton was campaigning the day after the primary. When she learned that Norton was trying to regain support that she had lost because of the tax issue, the woman said, "Poor lady . . . I had a husband like that too."

Norton said she does not need to regain her lost support in Ward 3 to win November's general election, in which she faces Republican Harry M. Singleton, a lawyer and former Reagan administration official, and independent George X Cure, a member of the Nation of Islam. She said she wants the support "to lead the city the way I want to from the Hill."

"The erosion of what was an excellent lead in Ward 3 was understandable," Norton said in an interview. "But I need to do all I can to show 3 that I am who they thought I was when they were giving me the lion's share of their votes."

Norton said that before the tax returns became an issue Friday, polls indicated that she was leading in Ward 3 with more than 40 percent of the vote. She received less than 15 percent Tuesday.

At noon yesterday, Norton appeared with Rep. Steny H. Hoyer (D-Md.) at a rally of federal workers at the Labor Department to protest the proposed furlough of federal workers.

In her brief remarks, Norton thanked the workers for their votes and for their support through her "crisis" of having to defend herself against tax allegations.

Speaking with reporters at the rally, Norton said the tax issue had severely cut into her support among white voters across the city and particularly in Ward 3, which she lost to D.C. Council member Betty Ann Kane (D-At Large). Ward 3 is predominantly white and contains many of the city's most affluent neighborhoods.

Norton's campaign strategists had counted on strong support in that ward and in other predominantly white areas of Northwest Washington. Her supporters said yesterday that Norton's support in predominantly black areas of Southeast Washington was strong enough to offset the loss.

Norton said in an interview yesterday that her accountants are reviewing her family's tax and finance records, and that as soon as that is complete, "we will answer any more questions."

She said she expects the review to be complete well before the Nov. 6 general election.

Staff writer Michael Abramowitz contributed to this report.