There's a sign on a wall identifying the structure as the Justine Morrell Building, housing an agency of the U.S. Department of Agriculture. But everybody calls it Annex 3.

A light industrial building dressed up as if for a party, it was added on like an afterthought in 1905 to the Auditor's Complex, the picturesque 19th-century pile at 14th Street and Independence Avenue SW whose high clock tower, oddly, never received a clock. The Auditor's Complex is undergoing a $12 million renovation -- clock still not included -- but Annex 3's future is uncertain.

Organizers of the Holocaust Memorial Museum, to be located immediately south of Annex 3 along Raoul Wallenberg Place (formerly 15th Street), and the General Services Administration want to tear it down. "We would like to make a marvelous entrance forecourt and plaza not only to the memorial and museum but also to the Auditor's Complex," explained Robert Mendelsohn, a member of the memorial council and vice chairman of its museum development committee.

This is a clear, straightforward, comprehensible goal, but it's a profound mistake.

Though by no means an architectural masterpiece, Annex 3, designed by James Knox Taylor, supervising architect of the Treasury from 1898 to 1912, is a likable piece and an unusual one, for Washington. Its principal elevation, stretching some 200 feet along Wallenberg Place, is strong, rhythmic. Underneath an emphatic cornice there's a march of brick piers and big, tall, arched windows -- this is a low, loftlike building with a touch of class.

"The aim," said architectural historian Richard Longstreth in testimony last December before the city's Historic Preservation Review Board, which unanimously opposed demolition, "was to give a building intended for utilitarian purposes a sense of order, dignity and grace commensurate with its pivotal site on the Mall."

Taylor accomplished this goal and then some. Despite its utilitarian purpose -- for many years the building housed presses for the Bureau of Engraving and Printing -- his building signaled an important shift in the architectural image of official, monumental Washington. Unlike its bravura, stone-faced, Beaux Arts successors, such as the District Building (1908) or the Pan American Union (1910), Annex 3 is faced in brick in deference to the Auditor's Complex, but like them it's a direct result of the City Beautiful Movement that was sweeping through the country (and especially through Washington) in the early 1900s.

Even so, one doesn't want to claim too much for the architecture -- it's a modest piece of history one can see. The overriding reasons to keep the building are urbanistic and symbolic, rather than architectural: Annex 3 anchors a prominent corner facing two of the nation's preeminent symbols, the Jefferson Memorial and the Washington Monument, and it provides an urban frame for architect James I. Freed's Holocaust Museum.

Make no mistake, Freed's design is fundamentally superb. Situated midblock between the romantic revival Auditor's Complex and the classic revival Bureau of Engraving and Printing, it mediates beautifully between the divergent styles and colors of the two.

Its east facade, with a splendid, limestone-faced semicylinder as a focal point, adds a rhythmic grace note to dreary 14th Street. Its north facade, facing the Auditor's Complex, is a stunning arrangement of brick towers. The west facade is in critical need of refinement, of adjustments in scale and detail to give more human dimension to its vast planes (Freed was told to try again last Thursday by the Commission of Fine Arts), but, focused upon the projecting, off-center hexagon of a Hall of Remembrance, it is an impressive stroke of monumental modernism.

And, crowning glory, the interior, skylit court promises to be an incredibly moving space -- tension-filled, suggesting in steel, brick and stone the terrifying applications of industrial logic at the dark heart of the Holocaust, and yet filled with life-giving light.

But there is a crucial, basic conflict between the proud modernism of this structure and its prominent site, a conflict that has been apparent from the very beginning of the project and that has played a central role in the long struggles among the client, the architects (Freed is the second) and the commissions reviewing the design. It is, simply, that the building wants to be two things at once -- a free-standing monument in a park and a good urban neighbor, a piece of the city facing the park.

When the Commission of Fine Arts last year insisted that the Hall of Remembrance, a somber exercise in solid geometry, be pushed back in order to line up with the long colonnade of the Bureau of Engraving and Printing, it was telling the memorial council and Freed to make the building less of the former, more of the latter. This right and proper advice was followed, with predictable, beneficial results.

The issue is precisely the same today: In proposing the demolition of Annex 3, the memorial council is insisting that the museum building be more a part of the Mall, less a part of the city. The symbolic implications of this way of thinking are sobering. If Annex 3 is destroyed to make a plaza-park, the Holocaust Memorial Museum and its striking Hall of Remembrance will forevermore exist in direct sightline with the Washington Monument as well as the Jefferson Memorial, and will rival both.

This, I submit, is wrong.