You hear it all over this city's old waterfront neighborhood of Queen Village - constant hammering.

It is the measured, precise sound of the middle class, industriously fixing up houses, tearing down old buildings, and changing the character of whole blocks.

The return of the middle class to inner-city areas, a familiar trend in Washington, D.C., has spread across the country, and many mayors and urban experts see it as something akin to divine salavation for blighted urban cores and depressed local budgets.

But the movement is not drawing unalloyed praise, as a recent Washington Post check has found here and in Baltimore, Portland Ore, and Columbus, Ohio. Increasingly, complaints are voiced about such side effects as skyrocketing housing costs and the displacement of poor, elderly and other longtime residents.

Geno Baroni, assistant secretary for neighborhoods at the Department of Housing and Urban Development, says, "There ought to be options allowing poor people to stay where they are. We have to ask who shapes and shares the burden of social change. Who gets hurt? Who pays the price?"

In Queen Village, Janice Theil got hurt.

Theil, her husband, Robert, and their two children were evicted last winter from an apartment across the street from the cafe where she works because the landlady wanted a place for her son to live.

The Theils looked for a rowhouse to buy in the neighborhood, where they had lived all their lives. But they could find nothing under $35,000 because developers and individual buyers, most of them young and affluent, have been investing here at a dizzying rate and driving up prices in the process.

So the Theils settled on a $33,000 house in an adjacent neighborhood, Pennsport, figuring it was all they could afford on her salary as a waitress and his as an airline baggage handler.

Now, when she drives to work at 6:30 each morning. She has trouble find a parking space because many of the new houses and apartments have been built without adequate parking.

But to Janice Theil and her sister, Doris Boyle, who runs Doris' Luncheonette, the most difficult aspect of Queen Village's rejuvenation is getting used to the newcomers.

"They don't like kids, noise, activity, playgrounds," Theil said. "All they like is dogs and plants. They're stuffed shirts. They wear [WORD ILLEGIBLE] and, would you believe it, gowns to parties, and they have what they call sitdown dinners. Why, when we have parties, we just put out the beer and the lunch meat."

"They're all longshoremen in this restaurant," Boyle added proudly. "No new people come in here. It's not nice enough for them."

Boyle said she and her husband, Daniel, intend to sell out and move to New Jersey because their property taxes went from $365 to [WORD ILLEGIBLE] in two years.

Richard and Kathy Conway fear the same thing will happen to them. He is a laborer at General Electric and she is a community organizer at the Queen Village Neighbors Association, and together they make $17,500 a year. Because their rent has tripled in recent years, they bought the shell of a row house for $25,000 and are renovating it themselves.

"We seem to be drowning in rent and mortgage payments," said Mrs. Conway. "We hope to move into our house at the end of the year, and maybe we'll be okay for a while. But deep down I feel we're not going to make it. Ultimately the taxes are going to drive us out."

Displacement has already changed the community.Last year it was estimated to have 7,000 peoples now the number is said to be 8,000. In 1970 about 40 percent were black now fewer than 25 percent are. People of eastern European descent once dominated the white population now it is more homogenous. Median income was about $9,000 in 1970, no one knows what it is now, but some say it has doubled.

More importantly, the fabric of life is changing. In the old days people would sit on their stoops and talk with their neighbors. The newcomers don't build stoops and they often put up iron gates for privacy.

In the old days people would not hesitate to discipline their neighbors' children if they were naughty - or feed them supper if they happened to be in the house at 6 p.m. Now there are fewer children and more people are strangers. Often the newcomers and the oldtimers do not speak to each other.

No one seems to know how much displacement is being caused by middle class investment in cities. But there are these indications that it is a growing issues:

The National Association of Neighborhoods which consists of 65 city wide neighborhood coalitions, held a national conference on the subject here last month NAN is writing a guidebook to help city officials avoid resident displacement, and it plans demonstrations in four cities later this year to dramatize the plight of those forced out of their homes.

A forthcoming study by the National Urban Coalition, an information and advocacy organization, found that displacement was reported as a widespread problem and that rehabilitated neighborhoods in 33 cities showed dramtic increases in economic status of residents.

A recent study by Phillip L. Clay, an urban expert at the Massachusetts Institute of Technology, found that some displacement had occurred in 46 of 57 neighborhoods that had undergone substantial middle class investment in the nation's 30 largest cities.

HUD, expressing growing concern over displacement, has begun awarding "innovative" grants to cities that are trying to cope with it. Last month it gave $160,000 to Savannah, Ga., to provide staff and architectural drawings for a group called Savannah Landmark, which is buying buildings in the city's Victorian District to rent to low- and moderate-income families. HUD has awarded other innovative grants to Columbus and Portland, which are also trying to help low-income families stay in upgrate neighborhoods.

While displacement in many Washington, D.C. neighborhood has racial overtones - whites are displacing blacks - that is not necessarily the case in other cities. In Baltimore, for example, blacks are being displaced in the Harwood and Barclay neighborhoods put poor whites are being forced out in Union Square. Here in Queen Village, poor whites and blacks are the victims.

Some urban experts doubt that heavy middle class investment in city slums is creating serious problems.

"You can't have everything," says urbanologist Richard P. Nathan of the Brookings Institution, a Washington research organization.

"Improving the tax base improving prospects that housing will be rehabilitated and that jobs will be created - these things are a tremendous benefit to a city. Sure, you ought to be concerned about relocating and assisting people who are adversely affected, but on balance, redevelopment is a plus," he says.

Columbus Mayor Tom Moody, president of the National League of Cities, said local governments should try to preserve as much of a mixed-income population as possible in neighborhoods being revived.

"But you cannot upgrade a community and keep all of the poor living there as they did before," he argued. "The poor will go to places just as bad or worse. It's necessary concomitant of the process . . .

"A few people will get hurt. But you have to let the private sector work where it can work . . . If the nation really cares about everyone having decent housing, then we should get national dollars to provide it."

Jeremy Alvarez, Philadelphia's area planner for 25 neighborhoods, including Queen Village, in South Philadelphia, said he does not think Queen Village homeowners have much to complain about.

"It can't be bad for homeowners to have property values go up," he said. "The renters are the real victims. The city doesn't have a good way to help them. We're torn on the displacement issue. We don't have a policy."

But Conrad Weiler has a policy and in various ways he and the Queen Village Neighbors Association, which he heads, have slowed the process that is forcing some oldtimesr to move.

The policy, simple, is to preserve as much of the old character of this onetime longshoreman's neighborhood as possible, to oppose poorly planned high-density housing and to keep the stores, shops and factories that provide the jobs and services the residents need.

Weiler, an associate professor of urban studies at Temple University, admits that as a middle-class newcomer who built a rowhouse here five years ago, "I'm part of the problem." But since he has taken up the cause of the working-class residents, the oldtimers count him as their own.

Weiler also heads the neighborhood association's zoning committee and, because the city's Zoning Board of Adjustment usually agrees with the committee's recommendations on zoning variances, he has established himself as the person to see for those planning to build or expand in Queen Village.

Four years ago a developer wanted support for a variance to put a 170-unit apartment building on a vacant tract. Weiler and his committee refused. Another developer proposed a 150-unit building. They refused again. Two years later a third developer wanted to build 55 townhouses. Weiler called them "tiny little boxes without adequate parking." Finally the developer proposed 37 houses with 40 parking spaces. Weiler reluctantly said yes.

"No poor person can afford these houses, he said. "The only thing we did was scale down the amount of development and shift it to families. It was the best we could get."

Weiler's committee has also talked a developer out of starting a restaurant and into opening a doctors' office, arguing that the elderly people here needed a general practitioner.

For all the association's efforts, this 6-by-12-block neighborhood is still a prime target for redevelopment, mainly by middle class whites from other parts of the city who cannot afford the restored townhouses of Society Hill, the fashionable area just to the north.

This year more than 250 units, mostly townhouses, will be added in Queen Village, and new houses that five years ago would not have cost more than $45,000 now sell for $100,000.

The Rev. James H. Tucker, minister of the black Phillips Temple Christian Methodist Church in the heart of Queen Village, said black tenants have been hard hit by the renewal and many have gone to public housing projects.

"When the new people came in, I just knew blacks were going to have to move out," he said. "We don't have that kind of money."

While the Queen Village Neighbors Association tries to slow displacement largely through zoning, other communities are trying other methods.

In Baltimore, for example, the St. Ambrose Housing Aid Center has helped 800 low-income black families become homeowners by urging landlords to sell to their tenants, helping them find financing, and even buying and repairing houses itself for nonprofit resale.

In Columbus, the Battelle Memorial Institute, a private research organization, has set up a management firm that is fixing the exteriors of houses it owns and offering them first to the tenants. A local neighborhood group has complained that the sprucing up will make the structures too expensive, but Mayor Moody says the city will use part of its new HUD grant to help the tenants buy.

In Portland, the city plans to buy and rehabilitate 40 substandard houses in various neighborhoods and make them into rental units for lower income people. But the program is not expected to be much help to the northwest part of town where middle class investment has more than doubled housing costs in the last two years, posing a displacement threat both to elderly residents in apartments and young adults in communal houses. There, the Northwest District Association, accordint to one member, has "just now begun to acknowledge that the problem exists."

In fighting displacement, the National Association of Neighborhoods says its goal is to promote diversity. "We genuinely feel that racially, economically and culturally diverse neighborhoods are the only ones that are real communities of justice," says NAN executive director Milton Kotler.

And Queen Village's Conrad Weiler raises the question: "When we quote unquote save the cities, which are mayors and pols are patting themselves of the back for, are we simply re-creating suburbia?"