The directors of a 2,500-member New Orleans group dedicated to furthering international trade and understanding have refused to invite U.S. Ambassador to the United Nations to address their group in his hometown.

The staff of International House suggested that Young be asked to speak to the organization at the end of next month, when he will be here to participate in a Tulane University program. However, the 105-member board voted Wednesday to delay action on the subject until after that time, effectively vetoing it.

The voice vote came after one member at the closed-door meeting reportedly said Young is "misrepresenting the United States" and is "unfit to hold office."

Because Young is black and because all but one of the International House board of directors are white, the charge of racism arose from New Orleans' black community, despite denials by those who attended the meeting.

In London, shortly after arriving from Nigeria, Young briefed British officials on his recent African talks.

When asked abou the New Orleans development, his press spokesman.Tom Offenberger, reported that Young "doesn't have any reaction to an invitation that he has not received."

International House, headquartered in a multi-storied building on a prestigious corner in New Orlean's central business district, is an organization that seeks out trade, honors foreign leaders and offers foreign language courses and overseas tours to its members.

Its board of directors is made up of commercial, professional and industrial leaders from the New Orleans metropolitan area.

In April, 1976, the organization honored Jordan's King Hussein and his late wife, Queen Alia. In May of that year, they honored the visiting French president, Valcry Gliscard d'Estaing.

Receptions planned for the near future include ones honoring ambassadors to the United States from Canada, Thailand, Zaire, and a Scandinavian country.

Young was born in New Orleans and began his education there. His parents still live in the city, where his father has been a dentist.

At a news conference today, the International House vote was denounced as "a manifestation of racism," and one participant came up with something more substantial: a reaction that could hit New Orleans in the pocketbook.

In a statement read at the news conference. Daniel Vincent, director of Total Community Action, Inc., in New Orleans, announced that his group is withdrawing sponsorship from two major conventions.

New Orleans, which depends heavily upon tourist and convention dollars, may this lose the 1978 conventon of the National Association of Community Developers, a 20,000-member group, and the 1977 meeting of the 3,000-member Louisana Association of Community Action Agency Directors.

Despite the controversy. Internatonal House board President G. Frank Purvis Jr. said his organizatino still wants to do something to honor Young.

"I think the members recognized this man as being one of the first ambassadors New Orleans has produced," he said. "Everyone in the city's proud of him."