Warren Richardson, who withdrew from consideration as assistant secretary of health and human services late Friday amid charges of anti-Semitism, yesterday said he was a victim of "guilt by association" and lashed out at the Connecticut congressman who first raised the allegations."

Richardson admitted that while he was general counsel of the arch-conservative Liberty Lobby he was "very disturbed from time to time" by the organization's anti-Semitic or racist activities.

Richardson maintained he was not anti-Semitic or racist, and claimed that while he was an official at Liberty Lobby, the "legislative issues" he worked on "were not anti-Semitic or racist." In his first interview since withdrawing his name, Richardson said HHS Secretary Richard S. Schweiker applied no pressure -- "none whatsoever," Richardson said.

But he was critical of Rep. Sam Gejdenson (D-Conn.), who led the opposition to his nomination. Richardson said he was upset that the 32-year-old congressman had never confronted him personally with the anti-Semitic charges before he took them to the press.

Friday, Richardson contacted Gejdenson's office to try to arrange a meeting with the freshman congressman, who is Jewish. Gejdenson could not be reached, but an aide told Richardson the congressman would contact him tomorrow.

Gejdenson, who at one point described Liberty Lobby as the "Ku Klux Klan without sheets," has received some "hate mail and phone calls" since his allegations against the organization were made public, according to an aide.

Richardson said he decided to pull his name from nomination because "I did not want to subject the president or the secretary to a prolonged political battle."

"I won the moral battle," he said, referring to Schweiker's statement, in which the secretary declared that a "careful review" had uncovered "no convincing evidence" Richardson had ever been anti-Jewish or racist.