The District government leases too much of its office space and should begin now to replace leased offices with buildings the government owns, a panel studying the District's operations and expenses said in its report yesterday.

The panel, dubbed the Rivlin Commission after its head, Alice M. Rivlin, called on the government to reduce its 3 million square feet of leased office space by two-thirds in the next decade. According to the commission, the District government also needs to consolidate its offices into fewer buildings. By taking those steps, the city could save up to $20 million a year by 1996, the commission said.

The issue of leased office space has been a controversial one for years. City employee groups and civic associations have contended that the District government squandered millions on leases of offices in the downtown area and throughout the city. The most recent controversy involved a $216 million lease for a temporary city hall to be constructed on North Capitol Street. The D.C. Council has recommended canceling the lease, but the contract with a group of developers that includes sports agent Richard A. Bennett Jr. and Washington Redskins wide receiver Art Monk is still in effect.

The report criticized D.C. Mayor Marion Barry's administration for failing to meet Barry's own goal of reducing the amount of leased space by 10 percent a year.

Barry, appearing at a news conference to accept the report, declined to respond to that criticism yesterday. Instead, Barry argued that the city had contributed to "economic development" through its decisions to build and lease space outside the central business dictrict.

In particular, the mayor said, the city's decision to open offices in Southeast Washington and to build the Frank D. Reeves Center at 14th and U streets in Northwest helped neighborhoods that otherwise could have languished.

In its report, the commission said the District owns 50 of the 187 buildings where it has offices. The average lease costs $11.50 a square foot, and the downtown office space costs about twice that amount. The panel noted that downtown office space in the District is among the most expensive in the country.

The amount spent on the leases has grown dramatically since 1985, the commission said, and will continue to grow significantly during the next five years. According to the report, the city now spends $41 million for leased space, up from $15.9 million in 1985. Because most of the leases contain automatic annual increases, the city will pay almost $61 million by fiscal year 1995.

Robert L. King, the city's real property manager, agreed yesterday that the District needs to reduce its leased space, and said the goal of cutting leases by two-thirds in 10 years is achievable.

King said there would be about a four- to five-year delay in seeing significant savings because of the time needed to select and buy or build new space.

Although most of the savings that would come from reducing leased space would not be realized for several years, the commission said, the city should start moving aggressively as leases expire.

It was not clear from the report how much flexibility the city has retained in its leases or whether canceling the leases would expose the city to costly penalties.

Much of the report was based on a study by Richard Hadsell, a former U.S. General Services Administration official who was responsible for federal office space in the area.