Richard E. Carver's leadership of the U.S. Conference of Mayors may be playing well in his hometown of Peoria, but it is bombing out in Detroit and Atlanta.

The result is a politically embarrassing dispute that thereatens to shatter the conference's carefully crafted public image of unity in defense of urban concerns.

The immediate cause of the conflict is Carver's reshuffling of committee chairmanships within the conference, the nationa lobbying group for cities with populations of more than 30,000.

Carver's critics say he is arbitrarily trying to shift the conference's traditional orientation from big cities in the East and Midwest to smaller cities in the Sunbelt.

Because there are no blacks among the conference's seven committee chairman, and because two black mayors, Coleman Young of Detroit and Maynard Jackson of Atlanta, where removed from chairmanships by Carver, some critics are also accusing him of racial insensitivity.

Carver is the fourth Republican president of the mayors' group in its 47-year history, and has publicyl disagreed with the heavily Democratic conference on policy matters.

He is also a candidate for the Senate and has been accused of using the conference presidency to further personal political goals.

Carver, who was in Chicago, could not be reached for comment yesterday.

John Gunther, the conference's executive director, who is caught in the middle of the battle, said the disputants will try to work out their differences Friday afternoon in a private Washington meeting.

"Some people are upset," Gunther acknowledged. "There have been some questions raised about the absence of black committee charimen."

Gunther said that under the conference's constitution the president has the authority to make committee assignments as he sees fit.

Four mayors, including Young and Jackson, were removed from chairmanships because they had already served on the conference's executive committee, Gunther said, emphasizing that he was only relaying Carver's explanation.

"Mayor Carver decided that the members of the 17-member executive committee should not also serve as chairmen of other committees . . . . He decided that the chairmanships should be spread around," Gunther said.

The problem is that the conference executive committee is viewed largely as the organization's housekeeping arm. The group's six standing committees frequently give their chairmen a base to become national spokesmen on aspects of urban policy.

"The mayors who were taken off the committees felt they had worked long and hard to get those committees estiblished, and they feel that to be dumped now just isn't right," Gunther said.

Young was removed from the Urban Economics Policy Committee, which he founded and chaired for the past three years. Jackson was taken off the committee on the arts, a panel he has chaired since founding it two years ago.

Also removed were Mayors Helen Boosalis of Lincoln, Neb., from the human development committee, and Lewis Murphy of Tucson from the transportation committee.

Young, Jackson and Boosalis are Democrats. Murphy is a Republican.

Young and Jackson were replaced by small-town Democrats. Boosalis was replaced by a Republican. Murphy, much to the chagrin of many mayors, was replaced by chicago Mayor Jane Byrne, a Democrat who has publicly criticized the mayors' conference as being weak and ineffective.