Alexandria's plan to save its financially troubled housing authority by selling an authority-owned housing project to the city and relocating its residents in Old Town is being attacked by local civic groups as hostile to the poor and potentially damaging to city property values.

The City Council last week unanimously approved a preliminary step that could raise an estimated $5.7 million for Alexandria to purchase the 6.1-acre John Roberts Public Housing project, located just west of downtown. That would give the Alexandria Redevelopment and Housing Authority money officials said it must have to keep from going broke.

The transaction also would insure city control of the private redevelopment of the John Roberts site, the real estate value of which is expected to soar after the nearby Braddock Road Metro Station opens in mid-to-late 1983.

"It the sale is not going to be the panacea to solve all our problems," said Angus T. Olson, the authority's executive director. "But if we hold the line on expenditures, we should be able to go on for five years."

But opposition to the plan -- which includes relocating John Roberts' 240 tenants into new and existing public housing in the middle of Old Town's waterfront along N. Fairfax Street -- is deepening. Opponents argued that the plan would burden the city with debt and work to keep low-income people in public housing ghettos and damage area property values.

In a recent letter to the council, the city's Chamber of Commerce wrote: "We believe the continuation of large numbers of public housing units in the same area leads to de facto segregation, to uneconomical maintenance and appearance problems, and a decline of general property values."

During the council's last meeting, more than two dozen people waited six hours to hear their spokesmen challenge the authority's plan to sell property to the city while it plans to rehabilitate its waterfront units with federal help.

"I think it's the same business of holding people in a certain location where everything is regimented and controlled," said John P. Flannery, who urged the council to drop its "foolish" proposal. Instead, Flannery, director of the Alexandria Taxpayers Alliance, urged the authority keep the 40-year-old John Roberts project and its tenants where they are.

If the authority needs money, alliance members suggested, it should sell its most valuable property--the four waterfront blocks of crumbling public housing called "The Berg," which is actually the Samuel Madden and George Parker projects, where the authority plans to move some of the John Roberts tenants.

Alliance members argued that money from the sale of the Madden and Parker projects should be invested and used to subsidize rent vouchers that would help defray the costs of private rental housing for those now in public housing.

That way, Flannery said, "these people could live like normal human beings." The crumbling project could then be razed and transformed by private developers into expensive town houses, returning millions of tax dollars to the city, Alliance members said.

"It's foolish to have the potentially most valuable property in the city sit tax free," said Alliance President John Williams.

"The Berg," with its worn red-brick facades and mostly unkept grounds, has been a source of irritaion to the mostly affluent white residents of the neat federal-period town houses near the largely black housing project.

Olson said the mechanisms for a public housing voucher system do not exist. Democratic council member Donald C. Casey called the voucher "a bus ticket out of town." Some community leaders charge opponents of the city plan with using the authority's financial woes -- hastened by drastic cuts in federal subsidies -- as an opportunity to dismantle public housing.

"The only motive is to get those black people out of The Berg , because they live on valuable property," said the council's only black member, Lionel R. Hope. "Some of those people have lived there for generations . . . They're there and would like to stay where they are."

Said Williams of those supporting the authority plan: "It's mind-boggling that they can't see right reason."