Brown's Bakery on Georgia Avenue NW always seems busy. Customers drop in to buy spice buns, plantain tarts, coco bread and copies of The Jamaica Weekly Gleaner. Lester Chen, 69, mans the cash register and generally, he says, keeps his opinions to himself.

But these days Chen, a native of Jamaica, is disturbed about the image others are receiving about his country and its people. Everybody, it seems, is talking about the violence, both locally and nationwide, involving Jamaican-born drug dealers and other criminals: Already area police had attributed about two dozen slayings since 1985 to Jamaican gangs, and then, most spectacularly, five Jamaican natives died a week ago in a drug-related slaying in Landover.

"It is very bad," said Chen, who is also of Chinese descent. "Sometimes I am afraid to say I am from Jamaica -- people automatically form opinions about you."

There is another side to the story of Jamaicans in the Washington area and it has to do with the hard-working residents who came here for an education or an opportunity not available back home -- where the unemployment rate is 22 percent and the average annual income is $1,130.

They are lawyers, university professors, restaurateurs, hotel workers and domestics. They live in Northwest Washington, in Takoma Park and Hyattsville, in Silver Spring and Alexandria. They stress that the recent crimes involve only a small portion of the estimated 10,000 Jamaicans living in the metropolitan region. Yet, they fear that publicity is stamping them and their country in a way that most Americans wouldn't tolerate if other ethnic groups were involved.

"What we find most distressing is, yes, these are crimes, they are dastardly crimes and they are crimes perpetrated by residents here in the U.S. But the label -- everybody's talking Jamaica," said Cecile Clayton, diplomatic minister of the Embassy of Jamaica. "It is interesting because nobody talks anymore about the Poles this or the Chinese that. This is a country made up of immigrants, right? But people are making the connection that the Jamaican people are the violent ones."

When Washington was created in 1791, Jamaicans were already living in the area, according to embassy officials, but community leaders said the largest group has come in the past 30 years.

Some of the earliest arrivals took jobs as domestics, but many others came here for a college education. The Howard University Alumni Association in Jamaica, for example, is a long-established organization, and the Howard staff has featured a large proportion of Jamaican natives, including the dean of the graduate school of arts and sciences, the former chairmen of the economics and zoology departments, and professors in political science and sociology.

"The Washington area is traditionally a college mecca for Jamaicans," said Jamaican native Bancroft Gordon, an all-America soccer star and Phi Beta Kappa at Howard who is now a lawyer with one of the region's largest law firms.

"We cannot dispute the facts about the recent crimes," Gordon, 30, said yesterday, "but there is a potential problem of xenophobia, of labeling people by their accents."

That, apparently, is already happening to an extent.

"Especially in the university halls, I hear the students complaining," said Dorith Grant-Wisdom, who left Jamaica in 1977 and is now an assistant professor of political science at Howard. "Caribbeans in general and Jamaicans in particular -- if you have an island accent, it's just assumed you're part of the typical criminal element."

In the past two years, the D.C. government has come under fire for its handling of the February 1986 drug crackdown called Operation Caribbean Cruise, which targeted alleged drug dealers from Caribbean countries and particularly Jamaican members of the Rastafarian religious sect. Last year, 33 people whose homes were raided during the ill-fated operation sued the D.C. government in U.S. District Court for "a vicious and unwarranted attack on innocent members of the black community," the suit said.

"We are concerned about the quickness of the media to attribute crimes to Jamaicans, to blame an expatriate group for this antisocial behavior -- not just the media, but the city government," said Ransford W. Palmer, 48, a Jamaican who teaches economics at Howard. "But when you look at the history of immigration here, pick out one group, the Italians. We know about the violence, the history of the Mafia. But we also know about {Lee} Iacocca and {Mario} Cuomo."

Still, it is the Jamaicans who are currently the topic of conversation, and Ethel Prussia said she is upset by the insinuations.

With her two sons, Prussia owns and operates Jamaica Joe, a homey, inexpensive restaurant in Silver Spring that features curried goat, jerk chicken and oxtail soup. The walls are covered with posters of bikini-clad women on Jamaican beaches -- "There's No Place Like Home -- Jamaica" -- and the green, black and yellow Jamaican flag. The menu, which warns patrons to use the Jamaican Hell Fire Sauce cautiously, also includes the full text of the Jamaican national anthem.

"I don't think the criminals are Jamaicans alone. I think they are a whole lot of people," said Prussia, who was a supervisor in a shoe plant before she left Jamaica 10 years ago for "adventure." "Most of us are quiet, nice people not involved in anything but hard work and pride for our country."

Her son, Gary Prussia, 30, said that some of his schoolmates have died here in drug-related shootings. At nightspots frequented by Jamaicans, everyone is talking about the violence, he said. Prussia has his own theory of what is happening.

"The older group of Jamaicans is looking for success," he said. "The younger group is also looking for success. They just want it the fast way."