In the midst of a well-kept neighborhood of renovated town houses and historic landmarks near Scott Circle stands the old Nigerian chancery, an eyesore, a health hazard and a nuisance, according to the complaints of its neighbors. Both the federal and District governments have expressed their concern over the situation.

The century-old five-story Victorian house at 16th and O streets NW has been empty since the Nigerian Embassy vacated it in 1974, although there have been several offers to rent or buy it as well as repeated efforts by civic and preservation groups and government officials to have it renovated and occupied.

The Nigerian government bought the building in 1965 and used it to house its information and visa application offices. The government now rents office space at 2215 M St., near its embassy at 2201 M Street NW.

"There is a large amount of rotting rubbish piled deep in the rear yard. Many windows are open both to the weather and to vandals. Vandals continue to strip the building of all of its architectural elements," wrote Charles Robertson, chairman of the preservation group, the Dupont Circle Conservancy, in a letter to the State Department two years ago. It is "not only an eyesore, but a serious health, fire and security problem for the neighbors," he added.

The former chancery's condition has deteriorated further since then. A Nigerian official who came to Washington in January lamented that "the building has been so much cannibalized."

For the last five years, attempts to persuade the Nigerians to restore and rent the building or board it up against vandals have been made by the D.C. Bureau of Community Hygiene, the Dupont Circle Citizens Association, Don't Tear It Down, the Dupont Historic Preservation Committee, the Joint Committee on Landmarks, and the Dupont Circle Advisory Neighborhood Commission (2B), among others. Several groups have offered to buy and renovate the building, including Cornell University, the law firm of Ingersoll & Bloch (whose office is across the street) and the Historic Development Coorporation of Philadelphia.

At a recent meeting about the building with State Department officials in the Office of Protocol, the Nigerian embassy's Head of Chancery, L.O.C. Agubuzu, reiterated his government's intention to retain possession of the property. According to a State Department official, he presented four options being considered for it, including a proposal to lease it out.

Another Nigerian official at the embassy disputed the rental proposal, however. "Why would we do that when we are already renting space on M Street?" he asked. "We will not rent it out to anyone else; we will use it for our annex offices."

A Nigerian embassy spokesman said his government in Lagos has been asked for funds to renovate the building, but so far has given no definite response.

Although the building has been secured several times, it has frequently been broken into. One neighbor said he took an unauthorized tour inside and found that the ornate mantles on the walls and the columns which once stood in the gothic library had been removed. Another curious neighbor said he saw discarded visa applications and other trash piled so high in one of the fourth-floor rooms that he and others complained to the fire department. Furniture had been removed and the building had some serious structural damage, said the man, who asked not to be identified.

Two weeks ago the five-story building was again secured, with sheets of plywood nailed over all its windows and doors against both vandals and the weather, at the request of the State Department Office of Protocol.

Ernest Harper, Chairman of the Joint Committee of Historic Landmarks, was encouraged by the progress. "At least the building is safe from the elements as well as from intruders," he said. He added that Nigeria is under a severe financial strain and may not be able to rehabilitate the structure.

Although the chancery building is owned by Nigeria, it is not considered foreign soil because it is located in the historic district of 16th Street, which runs from Scott Circle to Florida Avenue. It is subject to the district's building codes concerning historic preservation.

In April of 1979, the Dupont Circle ANC, which represents approximately 18,000 Dupont Circle residents, unanimously voted to prevent demolition of the building. Even without that vote, however, the building would have been protected from demolition by its historic status: since 1978, it has been listed on the National Register of Historic Places.

According to Richard Gookin, Associate Chief of Protocol at the State Department, the Nigerians a few years ago had planned to renovate the building and construct a taller facility around it for use by their growing staff. But in June of 1979, the city zoning board lowered the maximum height allowance from 90 to 65 feet. The Nigerians then tabled plans for both the construction and the renovation.

Built for $28,000 in 1881 as a private residence and remodeled in its present gothic style by George O. Totten in 1915, the mansion served in the 1930s as the Persian, and later the Venezuelan, legation. It was again a family home before Nigeria bought it in 1965.

Charles Robertson, who is also associate administrator of the American Museum of American Art, is skeptical that the Nigerians will do the work. "This is their story now," he said. "It has been their intention for seven years to fix it up. Maybe in another few years it will fall down . . . ."