The Vietnamese refugees who fled here from a falling country three years ago have been caught in the middle of a different war.

The conflict in this semitropical city and its nearby communities is not between communist and noncommunist. It is the long-running battle between black and white Americans; but many of the combatants in both racial camps have come to regard the South Vietnamese as enemies.

It is a peculiar struggle in which groups that should be allies are seemingly cast as opponents, and those that have been historical opponents are seen in many quarters as allies.

For example, there is the Urban League of Greater New Orleans, long the local champion of equal rights. The mostly black league has caused a major controversy here by suggesting that the new minority of South Vietnamese is growing far too rapidly and without due regard to the economic impact it is having on the city's predominantly poor population of blacks.

The immediate target of the league's criticism is the Associated Catholic Charities (ACC), an influential, predominantly white organization dedicated to helping the poor and which directed a portion of the resettlement of Vietnamese refugees in the area. ACC, in the past, has been loyal to the ideals of the Urban League.

Then there is Chalin O. Perez, a powerful white official of nearby Plaquenmines Parish (county), the scion of a wealthy family whose opposition to integration and "outsiders" is legend.

Perez, for once, agreed with blacks on something in accusing the federal government and the ACC of "dumping" South Vietnamese refugees into the area, with its French Catholic culture and its fishing heritage.

Perez announced last week that he will take steps that could lead to the departure of Vietnamese fishermen from his parish.

But the conflict swirling around the Vietnamese, about 8,000 of whom are living here and in adjacent towns, involves more than a listing of major players. It is a highly emotional thing, one that is largely a matter of perceptions, of long-held frustrations, and of what local Vietnamese leader Vu Huu Chuong called "dangerous misunderstandings."

Since May 1975, South Vietnamese refugees have flocked to the New Orleans area. An ACC official said his organization was directly responsible for bringing 2,100 refugees and that 5,400 other refugees came to the area on their own.

Many blacks here, like Dyan French Cole, a community activist, complained that the Vietnamese are being used by the whites to frustrate black ambitions for a better place in the local economic and social order.

"Every time you look around, somebody is coming here and being put ahead of the blacks who were born here," Cole said. "We were treated as property from the beginning, never as human beings. Now we are being told to step back and make room for the Vietnamese, just as we were told to step back and make room for the Cubans and everybody else the whites wanted to put ahead of us."

On the other hand, many whites complain that they have already "given in too much" to black demands. It is enough to have one minority group always knocking at your door without having to put up with another - in this case, the South Vietnamese refugees - doing the same thing, disgruntled whites say.

For example, Perez, president of the Plaquemines Parish Commission Council, expressed the feeling of many of his constituents when he said of the refugees: "What we have here is a group of people who have been dumped in our area who are totally unaccustomed to our ways, our manner of living, our mores and our laws."

The Plaquemines leader, who runs the parish from Pointe A La Hache (Point of The Hatchet), La., about 50 miles southeast of here, said he is upset by reports that Vietnamese fishermen are violating safety codes, showing nude in public on their boats, eating the pet cats and dogs of other fishermen, and flouting other social and local fishing industry practices.

He said last week that he will order parish officials to strictly enforce regulations that could drive the Vietnamese fishermen out of the parish by leaving them with no place to dock their boats.

"Just because they are Vietnamese doesn't mean that they are special cause," Perez said in announcing his plans. "It's their business [if they] go some place else to live up to reasonable standards," he said.

The problems here, real and imagined, are plentiful; and they tend to obscure the voices of those people, black and white, who say they welcome the Vietnamese to this area with no strings attached.

New Orleans, for instance, has what urban planners call a "distressed economy." Indeed, New Orleans is said by its mayor, Ernest N. Morial, and other urban planners to have "the most distressed economy" of any city its size in the United States."

There are an estimated 586,000 persons residing here, about 51 percent of them black. More than half the blacks, 56 percent, live at or below the poverty level. The city has a 7.6 percent overall unemployment rate.Unemployment in the black community is put at 22 percent for all adults, and at 45 percent for black teen-agers.

The two major industries here are the port and tourism. The local, state and federal governments provide about 40 percent of all available jobs in the city. There is a dearth of other gainful employment.

The public school system here, as in other cities, is in disarray. About 85 percent of its estimated 93,000 students are black, and most of them are poor. Housing for the low income is scarce, as evidenced by a public housing waiting list of 10,000 people, most of whom are black.

It is this competition for scarce resources, against a historical background of black-white friction, that is generating much of the discontent about the movement of Vietnamese to this area.

The Rev. Michael Haddad, executive director of the Associated Catholic Charities, which is considered the major force behind the South Vietnamese resettlement program here, said:

"The decision to admit more refugees is one that will affect only the poor in this country, and it won't affect the rich or the middle class. I know that's a problem, but the problem was here long before the Vietnamese came.

"The Vietnamese people are not the problem. And you just can't stop them from moving into the area because of all of the other problems that exist."

Clarence Barney, the executive director of the New Orleans Urban League, who has been embarrassingly portrayed in local newspaper editorials and cartoons are being in bed with the arch-conservative Perez, protests that he, too, is against excluding the refugees from this or any other community.

"Our argument is that in a community already troubled by a history of racism, and that has bad housing, bad education, and too few jobs, it is almost criminal to have a large number of new people come in and enter into unfair competition with the indigenous population that has suffered because of that history," Barney said.

"We ought to assure that the responsibility and obligation for helping the Vietnamese is carried out in such a way so as not to create additional burden and hardships for the people who are already here."

Barney acknowledged that his organization lost much local grace because of his position. But he said that a poll conducted by his group showed that about 92 percent of the community's black leaders agreed with the league's position, and thought that the local media criticism was unjustified.

National Urban League Executive Director Vernon Jordan has publicly adopted a hands-off policy in regard to the New Orleans dispute. But other national Urban League officials say that the organization is concerned about the economic impact of the refugee program on the nation's native poor, and that Barney's position is in accord with that concern.

Mayor Morial was out of town and could not be reached for comment. But other black leaders here say privately that it would be unwise for Morial or any other black politician to intervene in the conflict.

Daniel C. Thompson, a local university vice president and sociologist who also serves as president of the city's Coordinating Council of Black Leadership, explained:

"We have never seen blacks in this city so frustrated and mad as they are now about their economic position. There is a kind of reckless anger out there that wasn't there even in the 1960s, when, at least, there was some kind of hope."

Still, Thompson said he believes that much of the confusion and the charges that Vietnamese are displacing blacks and poor whites in jobs and housing could have been avoided.

"The problem is that all of the institutions in this city are either black or white or at least see themselves as either black or white, and there is no single institution around here to facilitate our movement into a desegregating society," he said.

Vu Huu Chuong, who serves as vice chairman of the New Orleans Area Vietnamese Committee, said he agrees with Thompson.

"We hope that the leaders in the black and in the white communities will try to understand the Vietnamese, and try to explain to us clearly what the laws and the customs so that we can make good citizens," he said.

He said that he would like to meet with Barney "and even with Mr. Perez" to discuss the refugees' position. Though his constituents have received considerable harassment from both blacks and whites, according to local officials here, Chuong said the Vietnamese community does not feel unwanted.

"We have many friends here among the blacks and whites. All of this [controversy] has come because of a very dangerous misunderstanding. We hope to resolve that. We will try."