President Carter's brother Billy escorted a high-level delegation from Libya through the gold-domed capitol of Georgia today despite a snub from the mayor and a cool, arm's-length handshake from the governor.

For the Libyans, the trip to Georgia and eight other states was the beginning of a drive to erase the regime's reputation as a bankroller of international terrorism.

But the first three days of a scheduled five-week visit have generated more controversy than goodwill, partly because of erratic behavior by Billy Carter, who the Libyans had hoped would win them increased acceptance in the United States.

Libya and its president, Col. Muammar Qaddafi, is best known to Americans as a supporter of terrorists ranging from the Japanese Red Army to the Palestine Liberation Organization. Qaddafi is also a leader of Arab opposition to Egyptian President Anwar Sadat's efforts to make peace with Israel.

But members of the Libyan delegation, led by Ahmed Shahati, the head of the country's foreign liaison office and a close adviser to Qaddafi, say Libya has moderated its extremely radical stance and now wishes better relations with the United States.

Billy Carter, who called the Libyans "some of the best friends 1 have in the world," says the United States must end its hostility toward the oilrich North African regime.

"There's a hell of a lot more Arabians than there is Jews," he has said frequently during the trip.

But his serious moments, in which he cites the massive trade deficit that the United States is running with Libya because oil imports cost billions of dollars more than the trickle of U.S. exports to the Tripoli government, have been overshadowed by his seeming inability to abandon the role of redneck cutup.

In Atlanta, at least, the most-talked-about phase of the visit so far seems to be a newspaper report that Billy Carter, while waiting for the Libyan delegation to arrive, got out of his limousine and relieved himself against the side of a building. Carter confirmed the report but said the area was secluded.

But despite the Billy Carter sideshow, the Libyans pressed ahead with a schedule that included lectures on the Georgia economy, a demonstration of new solar and nuclear energy techniques at Georgia Tech and a tour of the state capitol.

Atlanta Mayor Maynard Jackson refused to let Carter or the Libyans into his office. Gov. George Busbee met the delegation for about half an hour but an aide said later Busbee would do the same for any visiting dignitary, and the governor "does not want the impression left that he is soliciting the visit or that he is promoting export of Georgia products to Libya."

Shahati and his colleagues are scheduled to accompany Carter to his hometown of Plains Thursday. On Saturday, they are scheduled to leave for Miami. Other stops include Baton Rouge, La.; San Francisco, Sacramento, Los Angeles and San Diego, Salt Lake City, Detroit, Washington, New York and several stops in Wyoming.

Billy Carter will be the Libyans' escort only in Georgia. The visit to California is being coordinated by officials of the Occidental Oil Co., which does business in Libya.