At times it's downright scary.

At the very least, it's increasingly uncomfortable to walk past the expanding lineup of panhandlers around Dupont Circle, some of whom follow, threaten and shout obscenities at passersby near the Metro station, the drugstore and many Connecticut Avenue storefronts, residents complained at a community meeting last week.

The meeting was organized by the Dupont Circle Advisory Neighborhood Commission. Similar meetings are being held in neighborhoods across the city as citizens say more aggressive beggars and loiterers are making them afraid to walk on even well-lighted streets in high traffic areas.

Some neighborhoods are fighting back by encouraging people not to give money to the panhandlers. Attempts to remove beggars, whose numbers have increased with the District's soaring homeless rate, would be complicated by First Amendment protections, a representative of the American Civil Liberties Union said at the meeting. Two panhandlers even showed up to voice their views.

"You have two or three of them pushing cups in your face. They're making derogatory remarks," said a woman, recounting an episode by the drugstore. "People couldn't even go in the store."

"We are talking about young, able-bodied, capable of working, aggressive young people," said Jim McGrath, tenant association president of the Bay Street apartments on Massachusetts Avenue NW.

Two months ago, D.C. Police Chief Isaac Fulwood Jr. said he was concerned about an increasing number of complaints by women about harassment by panhandlers and was encouraging officers to respond by enforcing D.C. laws on disorderly conduct and indecent language.

Lt. Kevin P. Keegan, of the 3rd District, which patrols Dupont Circle, said officers do respond to complaints about panhandlers and said patrols have been increased in the area. Another 3rd District officer said that police policy is not so much to arrest panhandlers as to keep them moving.

"Enforcement has become a little stricter," Keegan said. "We're trying to help these people. At the same time, we're trying to curb any violence . . . that might take place."

But residents complained that the problem seemed worse, not better.

"People are scared. The police don't really want to be bothered," said Jim Crutchfield, 44, who lives at 17th and P streets NW. "There's a mental assault going on."

"Why do we have to put up with it?" a man shouted.

Jerry Holland, 43, a homeless man who brings in nearly $20 a day panhandling in the area, acknowledged the influx of aggressive panhandlers and said he doesn't like them any more than the rest of the neighborhood.

"If they don't give to me, I say 'God bless you' anyway," he said. "I don't bother anybody."

Holland, who said he took to the streets after his wife and child were killed in a car accident nine years ago, said the newcomers who harass and intimidate are "crack heads" and other substance abusers.

"It's just an ongoing problem," said Nancy Melville, Advisory Neighborhood Commission staff coordinator, who said panhandlers often congregate outside the commission's office at 1526 Connecticut Ave. NW. "{The police} tell us to call and they come and chase them away. But it's usually after {the panhandlers} throw up or are passed out."

A lawyer who would not give her name suggested that citizens organize against the beggars. "There's a great deal of resentment developing among homeowners in this neighborhood. An awful lot of people have guns," she said. "If we could somehow communicate that they {the beggars} are not safe in this neighborhood . . . . That's the solution and nobody will say it."

Arthur B. Spitzer, legal director for the local ACLU, said what panhandlers do is legal as long as they do not harass people. The District has no vagrancy laws.

"I don't think we can ban panhandling," he said. "People are entitled to stand on the sidewalks and engage you in conversation."

A member of the Capitol Hill Association of Merchants and Professionals told of efforts in his neighborhood to encourage people to hand the beggars cards that tell them where to get help..

"Let's not encourage panhandling on our streets," said James M. Didden. "We don't think it's a good use of your charitable dollars."

He said the number of people at local shelters and soup kitchens has risen since the program was adopted. The Business and Professional Association of Georgetown is planning to distribute similar cards.

After the meeting, David A. Maggi, a commissioner who chairs the Advisory Neighborhood committee on panhandling, said he would try to get a similar program started at Dupont Circle.

"It doesn't solve the problem," he said. "But at least it's the first step."