Owners of an office building in Silver Spring have agreed to reduce the size and height of their building after a four-year legal battle over whether the building violated the county zoning code.

The Montgomery County Department of Environmental Protection has agreed to reinstate the building permit for the structure, at 801 Wayne Ave., in the next few weeks if the owners take care of several remaining details. Construction was largely completed when the building permit was suspended in May 1983 after residents in the neighborhood filed suit charging that the building was too big and too tall.

The agreement from the owners came after a decision by the Maryland Court of Appeals last January that the building did violate the county zoning code. The court said new plans for its alteration should be submitted to the county for approval.

Representatives of Permanent Financial Corp., which owns the building, met this week with county officials and members of the Seven Oaks-Evanswood Citizens Group to hammer out the last details of their agreement.

The owners have agreed to remove a fifth-floor penthouse, push back the front wall of the building and lop off several feet from the edge of fourth floor so that it will not protrude beyond the edge of the third floor.

When construction stopped on the 54-foot-high building in 1983, it was found to be 18 feet above allowable height restrictions and had 26 percent more floor space than allowed under zoning density limits. It was also built too close to the lot line.

Because the county had approved the height, however, the court said it could not require the building to be reduced in height. The modifications proposed by the owners will bring the building height down to 40 feet.

Moving the wall at the base of the building was anticipated to be a big problem and expense for Permanent Financial, which was acquired last August by Citicorp. County environment protection chief John Menke agreed, however, to let the columns that support the building remain where they are if the glass wall is moved back two feet and four inches.

Karen S. Burns, a member of Seven Oaks citizens committee, suggested that the building code did not allow Permanent to retain the supporting columns beyond the perimeter of the building, but county officials said the columns were acceptable under the code.

Charles Hall of Ward/Hall Associates, architects for the remodeling of the building, said, "We tried to make changes that would have a positive impact on the building."

The stucco-faced building also will be painted a "warm gray," Hall said, to make it more compatible with a church next door. Joseph Blocher, Permanent's attorney, said the owners also will consider planting trees on both street sides of the building.

In closing the meeting, Menke told the architect and the owner's representatives that when the case had first come up four years ago, he thought that 801 Wayne Ave. was one of the "ugliest damn buildings I ever saw in my life."

"I think you have gone out of your way to make a better building and a better neighborhood and you deserve credit for that," Menke said.

Members of the citizens group, however, said that they were still dissatisfied.

"Well, it's still an ugly building," said Kathleen Charner, one of the parties to the original lawsuit. "I'd like to see something that blends with the surrounding neighborhood better." Regarding the additional landscaping the group had requested, and which Blocher had promised would be "beefed up" by the developer, Charner said, "Why should I trust them now?"

However, once the developer has met the building codes, said Karen Burns, a member of the group's executive committee, there is little it can do. "This could have happened in anybody's neighborhood."