Negotiations on turning Rockville Mall into an office complex have collapsed, and the pension system that holds the mortgage is preparing foreclosure proceedings that could shut down the mall.

Spokesmen for the City of Rockville and the New York State Teacher's Retirement System confirmed the latest developments, but officials of Rockville Redevelopment Associates could not be reached for comment.

The retirement system has not received a payment on the original $8.5 million mortgage since November 1978, according to Harry Dubrin, the fund's assistant executive director. The mortgage was issued in April 1972, and the outstanding balance is nearly $8 million. "We have to seek legal remedies," he said.

With about 350,000 square feet of retail space--and now only six tenants--the mall has been an albatross for the city and for a series of retailers and management firms since it was opened 20 years ago amid talk of reviving the downtown of Rockville, the county seat. The mall is near the intersection of Maryland routes 355 and 28.

A Bethesda development company, Eisinger-Kilbane & Associates, was on the verge of buying the mall and converting it into an office building when Rockville city officials became disenchanted with the terms of the firm's takeover of the underground parking facility below the mall.

"I can't conceive of what the city is thinking," said Roger Eisinger of the firm. "We thought we had a deal."

Under the proposed sale, the company was to lease the garage and its approximately 1,500 parking spaces for 99 years for $25 million over the first 45 years of the lease before switching to a market rent arrangement. "They decided they do not want to lease the garage," Eisinger said.

"The city could not go along with all they wanted concerning the garage," said Larry Blick, Rockville city manager. Blick would not elaborate.

But Blick and Eisinger agree with assessments by real estate analysts that the mall's future lies in converting the facilty into offices with a small retail facility principally designed to serve those tenants. The shopping area would utilize no more than about 30,000 square feet, under the Eisinger plan.

"The analysts say there isn't a sufficient market here to support a major retail center," Blick said. "We believe it has to be restructured."