A low-key movement is under way on Capitol Hill to find a way to remove J. Edgar Hoover's name from the FBI headquarters building.

Rep. Don Edwards (D.Calif.), an ex-agent and chairman of the House Judiciary subcommittee that oversees the bureau, said in a recent telephone interview that he thinks naming the building after Hoover is "inappropriate and that someday, quietly, it ought to be changed."

There also has been some discussion of the matter in the Senate, sources there said, although no one in either chamber seems ready to sponsor a bill to repeal the law that named the $120 million building on Pennsylvania Avenue after Hoover, the long-time director who died in 1972.

Any move to strip Hoover's name from the building is expected to meet with heated opposition from groups such as the Society of Former Special Agents of the FBI, which is holding its annual convention in Washington.

The society is scheduled to unveil a statue and plaque honoring Hoover and all FBI agents at the building this afternoon. Charles Stanley, president of the society, said yesterday that congressional action to change the name of the building "would be the mos ridiculous thing this country has ever done. I spent 33 years in the FBI and worked for Mr. Hoover, and there has never been a more patriotic or dedicated American than he was."

Edwards said he was exploring ways to change the name of the building because of several events. He said he frequently is asked why the building honors Hoover while revelations of Hoover-inspired abuses of citizens' rights continue.

"There's been a lot of indignation over the Jean Seberg thing," Edwards said, referring to recent disclosures that the FBI planted rumors in the press that the actress was pregnant by a Black Panther Party leader.

"All of the older Hoover generation is gone. There's a breath of fresh air over there" at the FBI, Edwards added. "So it seems something of an anachronism to have it named after Hoover."

FBI officials said they had heard nothing about the discussion in Congress to rename the building. And Edwards said he was in no hurry to push the idea. "We're busy with things a lot more significant," he said. "This [changing the name] is so emotionally charged, it would be a shame to have it get in the way of other issues."

Edwards' subcommittee, for instance, is to hold hearings soon on a proposed legislative charter for the FBI, designed to prevent a reccurrence of past abuses.

Debate about changing the building's name has been confined largely to "letters-to-the-editor" columns, editorial cartoons and occasional swipes by columnists.

The building was completed in 1975. Rep. Gilbert Bude (R-Md.) introduced a bill that year to remove Hoover's name from the edifice after reports that Hoover had directed a smear campaign against civil rights leader Martin Luther King Jr.

When informed of the latest thinking in Congress, Larry Heim, another member of the society of former agents, said "Don't you believe it. We've got a personal commitment from Director [William H.] Webster that he'll oppose any such move. This is just the current wave. It too will pass."