The Rockefeller Center Development Corp. said yesterday it was withdrawing from competition for the Metro Center redevelopment project because it fears Washington's overbuilt office market has made the project unprofitable.

"It is our perception that there has been a very marked decline in demand by potential users" for office space, said James Reed, a spokesman for the real estate company controlled by the Rockefeller family. The demand has softened both in Washington and across the country, he said.

The Rockefeller decision was another telling sign that the city's office building market faces serious trouble because of a glut of empty space now available or already under construction.

The recession and the federal government's deregulation efforts have nearly flattened growth in demand for office space by law firms, trade associations and accounting firms, which had fueled a years-long office space boom here.

An unprecedented 3 million to 4 million square feet of new office space will be built in Washington before the end of the year, and 4 million more is under construction for delivery in 1983, meaning an excess of perhaps 2 million square feet, according to real estate sources.

The team headed by Rockefeller was among three groups competing for the right to develop the 3.7 acre Metro Center site, located on the north side of G Street between 11th and 13th and atop one of the city's busiest subway stations.

The project was previously awarded to developers Oliver T. Carr and Theodore Hagans, but the city's urban renewal agency reopened the competition when it could not agree with Carr and Hagans on a price for the land.

The Rockefeller interests wrote their partners -- Washington's Western Development Corporation and five minority partners -- last week of their decision to pull out. The other partners were not consulted before the decision, sources said.

Western and the five minority partners, consultants Marie Barksdale and John Clyburn, attorneys David Wilmot and Ruby McZier and congressional aide Carolyn Jordan, issued a statement late yesterday saying they would remain in the competition.

But the withdrawal means that Western, which has tried unsuccessfully three times to obtain one of the city's potentially lucrative redevelopment sites, and the minority partners will either have to find a new big-money partner or discover a way to handle the project alone.

James O. Kerr, development administrator for the city housing department, said Washigton's urban renewal agency, which owns the site, was unaware of the Rockefeller decision.

It was unclear yesterday whether the urban renewal agency will allow Western and its five partners to continue alone or to bring in another big-money partner. When that agency reopened the Metro Center competition, it ruled that only original bidders could compete.

As additional reasons for their withdrawal, Rockefeller officials cited the court fight over the city's efforts to replace Carr and Hagans as project developers and the city's delay in choosing new developers.

In July, a D.C. Superior Court judge, responding to a suit from Carr and Hagans, blocked the Redevelopment Land Agency from selecting a new developer for Metro Center, a decision that had been scheduled to be made the following day. In issuing that preliminary injunction, Judge Sylvia Bacon said that the city acted improperly when it revoked Carr and Hagan's rights.

A trial on that suit is scheduled for November.