It's almost impossible to tear down a 100-year-old building in historic Alexandria, but a 96-year-old building is fair game.

And that's just what preservationists fear will befall the ornate Victorian May House on the city's main street as early as this morning. Late yesterday afternoon, the city's building inspector granted a demolition permit to the Professional Insurance Agents, a society next door that plans to tear down the house to expand its offices.

"The house is irreplaceable," said local historian William Seale, one of the preservationists trying to save it. "It was very modern for its time, and there is nothing else like it in the city."

But because the May House falls four years short of being legally historic under Alexandria law, Seale and company have been reduced to pleading with the society for a 60-day extension in which to try to work out a compromise that would save the two-story, red-brick building at 418 N. Washington St.

"My great fear is that they will knock it down this weekend, before anyone knows about it," Seale said. "The one thing that's been learned in all these years of controversy over old buildings is that the controversy ends when the building is torn down."

Trevor White, president of the insurance society, could not be reached for comment yesterday. "I talked to him this week and he said he was sorry but the decision was made," said Seale. "He said his board had bought the property on the condition that it be torn down. I asked him, 'Why don't you move to Crystal City?' "

"Some people who come to this town because of its charm and quaintness and then set out to destroy the very thing they profess to admire," said William Warwick, chairman of the Alexandria Board of Architectural Review.

The May House was built in 1886 and is a copy of a casino built around the same time in Short Hills, N.J. The New York firm of McKim, Meade and White designed the building. Stanford White was a reknowned turn-of-the-century architect memorialized recently in the film "Ragtime," in which he was portrayed by author Norman Mailer.

With its elaborate copper moldings, large arched windows and the tower that adorns its side, Seale said, the house is of the neo-Queen Anne period and was considered avant garde in its day.

"It's the equivalent of the wild contemporaries you see in McLean," Seale said. "My God, it's a wonderful old house." He contends the house is one of the few "real" pieces of architecture in the city. "It's very unusual for a provincial Virginia town."

Nonetheless, demolition can begin any time, according to the city building inspector's office. "The sewer's been capped, and as far as the structure goes, it's not a tough one to tear down," said city engineer Uwe Hinz. "They could probably do it in a day or so. Four dump-truck loads, I'd imagine."