It has been a visual point of reference for Dupont Circle residents for nearly a century, first as an exclusive house, and then as a series of restaurants. And now that the Fraser mansion has been restored to its 1920s grandeur, housing the Four Ways Restaurant, it has been the center of a longstanding local controversy.

International restaurateur Walter Sommer bought the property at 20th and R streets NW for about $2 million in 1981, and, to the delight of the neighborhood, poured another $3 million into the restoration of the mansion, scarred by years of repainting and decidedly unhistoric remodeling. For an area of fashionable shops and old-fashioned apartment buildings and town houses, where the traditional is as highly valued as the trendy, Sommer was a dream come true.

"I love the restaurant the way they've fixed it up, and what they've done to the house to restore it," said Mary Lib Craven, a lifelong Dupont Circle resident who saw the mansion through its various incarnations as the Parrot Tea Room in the 1920s, the Golden Parrot, and then as such contemporary nightclubs as Larry Brown's and Sagittarius in the 1970s.

For some, however, the thrill of the Fraser mansion's renewal is gone. Sommer has been trying to gain approval from the city's Zoning Commission to build a seven-story apartment building in the parking lot next door to the mansion. The eight-story apartment building on the other side of the parking lot, the Rodney, overlooks the mansion and beyond to Connecticut Avenue with a facade filled with balconies.

The design for the proposed building was approved by the Historic Preservation Review Board and the D.C. Office of Planning. The Department of Public Works has declared that the plans, which include underground parking for the apartments and the restaurant, would not increase parking or traffic problems in the area.

Sommer's only hurdle remains the Zoning Commission, which voted 4 to 0 in July against the zoning change. Sommer is asking the commission to reconsider the proposal with a more precisely worded covenant restricting the use of the new building to apartments or condominiums.

The project has been stalled, in part because of a vociferous and well-organized outcry from the neighborhood.

The Fraser mansion was built for former congressman George Fraser in 1890.

"It was such a dark and rich looking place," said Mary Tregillis, who has lived in the area since the 1950s, remembering the Golden Parrot Restaurant, which occupied the mansion after World War II until 1974. "The windows were all covered, there was a light on the porch, and a doorman in uniform, and I was always afraid to go in because I thought I couldn't afford to eat there."

When the Golden Parrot Restaurant gave way to a string of less than posh restaurants and clubs in the '70s, the prices became less forbidding, but the building did not.

Tregillis recalled the transformation of the building as the trappings of Booeymonger, the last occupant, were being peeled from the building. "I was so surprised when they were stripping the paint off the front porch, and lo and behold there was sandstone."

Sommer, who went into business on his own after leaving Princess Hotels International as president and chief executive officer in 1976, said he is frustrated by the zoning system, as well as by the vehemence of the opposition.

"I have taken every conceivable approach to restore this building to its highest glory, and I am proud of what we've done," Sommer said. "They say Walter Sommer is a nice fellow, we like him as a neighbor, but he should build a Japanese water garden in the back {in the parking lot}. By that time I nearly had a heart attack."

For Sommer, the dispute is a matter of whether the mansion will be permitted to support itself. "It has to stand on its own two feet, and it mustn't pay $150,000 to keep the parking lot," he said, citing the yearly cost of financing the parking lot. "For $300,000 we can have the same 25 spaces {underground} without the noise and smell."

The Fraser mansion is one of those buildings whose history and esthetics have helped the Dupont Circle neighborhood remain intact in the face of the crushing pressure of urban development.

Duff Gilfont, who has lived in the same apartment nearby for 30 years, is disturbed by Sommer's proposal. "It would be such a blight to this area, so I'm very glad the commission saw it our way, especially because there would have been so many people inconvenienced by it."

The property has always been zoned residential with variances to allow for a restaurant on the premises, but Sommer says the variances restrict him unfairly. He has had to go to the zoning adjustment board four times, a process he says is costly and time consuming, for approval of each change he wanted to make in his business. In addition, without commercial zoning he said he is unable to get "a realistic commercial loan" on the restaurant to cover maintenance costs.

"I am optimistic; if I were not, I would not be here," Sommer insisted. "But I am very disgusted with what has happened to me here. I have restored buildings in England, Victorian buildings, I restored buildings in Bermuda. But it's become a political issue, that's what it boils down to."