PORTSMOUTH, VA., JULY 15 -- There was a day when this struggling city looked upon James G. Holley III as a human bridge between once-warring blacks and whites, the kind of politician whose 1984 election as Portsmouth's first black mayor meant that bad times were history and the future was bright.

But that was before Holley lashed out one night at many of the same black parents who helped elect him, before reports about the $33,000 tab for his trips, before reports about hundreds of long-distance telephone calls he made at taxpayer expense.

If the good feelings about the Howard University graduate and former civil rights activist ebbed before this week, they have vanished now, after the disclosure this week that Holley's fingerprints were found on three of 30 pieces of hate mail sent last year to black officials and community leaders. The revelation not only has destroyed this city's faith in its mayor, but it also has shaken the city's confidence in itself.

"The people of Portsmouth have a lot of pain," said E.G. (Tip) Corprew Jr., who was elected to the City Council with the same black support that sent Holley to the mayor's office. Corprew, a 46-year-old mortician, said that "before the mayor was implicated in these actions, we had established a sense of pride, a look-forward attitude. For this to come, I won't say it stripped us of that, it stopped us dead in our tracks."

Holley has emphatically denied sending the hate mail -- much of it copies of newspaper articles on which racist, lewd or threatening remarks were scrawled -- but he also has declined so far to provide a handwriting sample to the prosecutor, who is looking into the letter campaign.

In addition to finding Holley's fingerprints, investigators found that the copies had been made on the sixth floor of City Hall, where the mayor's office is, and that the mail was sent in envelopes bearing addresses typed at his direction by his former secretary.

"People who have known me through the years would certainly know that these types of letters would be out of character for me," Holley said at one point this week. "I personally condemn anyone who would be involved in any such activities."

Nonetheless, this community in the heart of the Hampton Roads port region is not buying it. On Monday night, Corprew and the five other City Council members -- one black and four whites -- asked Holley to resign. The mayor refused and angry residents quickly launched a recall petition to drive him from office.

"Holley should resign" was the headline over the lead editorial in this morning's Norfolk Virginian-Pilot, which has broken much of the Holley story. "Mayor Holley's usefulness as an elected official has ended," the editorial said. "Portsmouth's interests would be best served if Mayor Holley exited, quietly and quickly, from the mayor's office."

Political and community leaders here doubt that Holley will leave any time soon, but several of them said in interviews that his continued presence could hurt the city they believe to be on the verge of realizing its dream of prosperity and pride.

{Holley could not be reached for comment today, but his wife Mary suggested to a reporter who visited their west Portsmouth home that the controversy may soon blow over. "It's just politics," she said.}

The mayor's troubles could not have come at a worse time for Portsmouth, which since the demolition of its run-down waterfront strip of bars and tattoo parlors in the 1970s has attracted enough private investment to stage a comeback of sorts.

A downtown district of restored antique houses -- the largest such concentration between Alexandria and Charleston, S.C. -- is a gem, and a magnet to law firms, banks and other "clean" industries.

At the same time, the city's story is unfinished and pockets of poverty remain. Last year, the median household income in Portsmouth was $21,348 -- putting the city in 80th place out of 136 cities and counties ranked by a research arm of the University of Virginia. Fairfax County was first, with a median income level of $45,639.

Despite the relative poverty, residents here were optimistic until recently. "I thought we were on the right road to everything," said Helen V. Davis, a city School Board member who attended the city's former black-only high school with Holley and worked for him on his City Council and mayoral campaigns.

Now, she added, "This city is disrupted. I'm in shock about everything. I'm in limbo. I'm numb."

Former mayor Richard J. Davis said Holley's refusal to resign could one day overshadow his considerable achievements in civil rights.

"You know, he helped integrate the public library, and when the local papers were still running 'Colored Notes' in the 1960s, he got 'em to stop," Davis said.

"I thought we had turned Portsmouth into a progressive leader for all of Tidewater, but this thing has cast a pall over us," Davis added. "It's a tough day for the city, but I guess we'll survive it. We've survived worse."