NEW YORK, DEC. 7 -- In a case that has become a symbol of rampant overdevelopment in Manhattan, a state judge ruled today that city officials had struck an illegal deal with developer Mortimer B. Zuckerman by selling him the right to increase the size of a proposed building at Columbus Circle for $57 million.

New York City cannot make "a 'cash sale' of a zoning bonus," Manhattan Supreme Court Justice Edward H. Lehner ruled. He said this amounted to "an illegal payment . . . . Government may not place itself in the position of reaping a cash premium because one of its agencies bestows a cash benefit upon a developer."

As a practical matter, Zuckerman's plans for 68- and 58-story towers at Columbus Circle were killed last week when Salomon Bros., the investment banking firm that was his major tenant and financial partner, pulled out of the venture, citing staff layoffs.

Zuckerman said today he is proceeding with a scaled-down version of the project, which would require a reduction in the record $455 million price he had negotiated for the city-owned site.

The project, which would replace the vacant New York Coliseum, has been the target of a high-powered publicity campaign, with celebrities such as Jacqueline Kennedy Onassis and Bill Moyers charging that it would mar the Upper West Side neighborhood and cast imposing shadows across Central Park.

Kent L. Barwick, president of the Municipal Art Society, which filed the suit, said the ruling "provides a measure of protection against the sale of city property to the highest bidder, regardless of the consequences. The real message here is that the city should be balancing interests, rather than acting as an overzealous developer."

Zuckerman, publisher of U.S. News & World Report and The Atlantic, defended the city's auctioning of the site, saying, "A lot of buildings have been built with zoning bonuses quite similar to what was done here."

But Zuckerman acknowledged that the project had "become one of the examples of what people perceive to be overdevelopment." Under the revised plan, he said, "we'll reduce the building substantially . . . and try to make sure the building works more in tune with the community's interests."

New York Mayor Edward I. Koch said the project's opponents had won "a temporary victory" and that the city would appeal. He said the city would suffer "substantial harm" if deprived of the $455 million purchase price.

"If the project is not built, there will be fewer policemen, fewer sanitation workers, fewer teachers and substantially fewer dollars for transit," Koch said. "Thousands of municipal jobs would be at risk."

Zuckerman's project and Television City, a 13-block West Side development planned by Donald J. Trump, appear to have galvanized public opinion here against the onslaught of skyscrapers that block the sky and add to traffic and congestion.

When Zuckerman beat 15 other developers during a 1985 auction, $57 million of his bid was for a bonus enabling him to erect a building 20 percent larger than zoning laws normally would allow. The proceeds were to be used by the Metropolitan Transportation Authority, which owns the site, for subway construction and for maintaining the $1 subway fare.

Zuckerman's firm, Boston Properties, agreed to contribute another $40 million to upgrade the nearby subway station at 59th Street and Broadway.

While the city frequently negotiates zoning bonuses with developers after the fact, Lehner adopted the Municipal Art Society's argument that the city had perverted the zoning process by including the 20 percent bonus from the start and requiring developers to raise their bids accordingly.

The expected reduction in Zuckerman's purchase price has created an immediate headache for Koch, who had included about half the proceeds in next year's budget. It could also lead to legal challenges by some of the losing bidders.

Zuckerman said the project, which originally included about 2 million square feet for Salomon Bros.' trading floors, would be changed to increase the proportion of luxury housing. The brokerage's withdrawal is part of retrenchment and layoffs by major Wall Street firms after the Oct. stock market collapse.