Sydney Rittenberg, man of mystery, the last American to serve time in a Chinese prison, winked from behind his thick glasses. "I expect to see myself in the rogues gallery someday," he said with a grin.

A secretive adviser to the Chinese government since the 1940s, released only months ago after 9 1/2 years in a Peking jail for an alleged ultraleftist plot against the state, Rittenberg was the last man one would expect to find on a carefree vacation trip through southern China.

Yet there he was, in shorts and sandals, enjoying the sights along the Li River and chatting easily about his life as the most notorious of the "foreign experts," the men and women who were drawn to China and its Communist revolution decades ago and who never went back home.

Rittenberg, 56, represents perhaps the oddest and most intriguing chapter in the century-old story of Americans and their love affair with China. He came here with the U.S. Army Signal Corps in 1945, then stayed on to work for the new Communist government and become what some called "perhaps the finest Chinese-English interpreter in the world."

He may have also been, for a while, the most powerful foreigner ever to work in Mao Tse-tung's government, a fact that came to irk some Chinese.For a few months in 1967, Red Guard publications and Rittenberg friends said, this son of a prominent Charleston, S.C., family actually ran China's principal radio broadcasting center, which regularly and vehemently denounced the land of his birth.

Shortly after that, Rittenberg's life fell apart.

In a confusing series of events that he says he still does not completely understand, he was charged both with being an American spy and with helping a campaign by ultraleftist Chinese to overthrow Premier Chou En-Lai. His apartment, once decorated with Ming Dynasty antiques, was sealed and he was thrown in jail.

Long after Peking had released its few actual American spies - such as the CIA agent John Downey - in the aftermath of President Nixon's 1972 visit, Rittenberg remainded in prison. Only late last year was he allowed to return to a quiet life in Peking with his Chinese wife and four children, working as a translator-editor and remaining as much a mystery as ever.

For 30 years he has avoided Western journalists, but during a chance encounter in this resort town he seemed to relax, unveiling a courtly South Carolina charm and accent and yielding to a desire to explain what had happened to him.

It was misunderstanding and a frame-up, he said, as the tour boat we were on cruised down the river. The "Gang of Four," the dogmatic faction whose purge in late 1976 led to his eventual release, had falsely blackened his name, he said. "People don't understand what that group was like. They were fascists. They would take anything you said and use it against you," he said.

By all accounts, Rittenberg had been active in the Culture Revolution campaign to weed out wrongheaded Chinese bureaucrats - a campaign organized by Communist Party Chairman Mao and pursued vigorously by his wife, ChiChiang Ching. But Chiang Ching, purged in 1976 as part of the Gang of Four, had turned against Rittenberg when he would not cooperate with her attempt to unseat Premier Chou, Rittenberg said. Perhaps she was just a victim herself of people in Peking opposed to woman leaders, someone suggested. "She wasn't a woman, she was a bitch," Rittenberg replied.

"They tried to get us to participate in putting out some stuff against Chou," he said. "We absolutely refused. So what they did was do themselves what they had asked us to do, and then they pinned it on us."

He said he had not read much of what had been written about him abroad, but he seemed fascinated and annoyed by some fo the tales that had filtered back to Peking. He asked about Washington journalist Warren Unna, a friend from army days, who had written a story in The Washington Post about his memories of Rittenberg after Rittenberg's arrest was announced in Peking in 1968.

"I hear he wrote that I was an imposter, that the real Sydney Rittenberg was dead," Rittenberg said. Actually, Unna had simply reported that Rittenberg's family in Charleston had apparently written him off as if he were dead.

Rittenberg had heard about, or perhaps read, a 1975 book by two Americans who had taught in Peking in the late 1960s and whom he had known well, David and Nancy Milton. Rittenberg's rise and fall provided much of the drama of the Miltons' book, "The Wind Will Not Subside."

They said Rittenberg was the man selected as the Chinese governments "go-between in managing a community of idiosyncratic foreigners," performing the same service a few Chinese-speaking Jesuits had for the Chinese emperors in the 17th century.

"Peking's entire foreign community" leaned on Rittenberg, a prominent figure in brown corduroy suit and disheveled tie, "for information, expertise, and wisdom regarding China," the Miltons said. His luncheon table at the Friendship Hotel was always full; he was "a man with an undefined mystique of power bestowed by the Chinese," they said.

Rittenberg said he finds such description cloying, and likes even less the Miltons' explanations of his fall. "My former friends, David and Nancy Milton, said I was involved in a conspiracy against Chou En-lai, which was absolutely untrue," he said.

When Rittenberg was finally released last fall, he was telephoned by the London Daily Telegraph correspondent in Peking and by an Associated Press reporter in Hong Kong, but declined to say much. "They asked where I was and what the conditions had been," Rittenberg recalled. "I just said the conditions had been good. They asked if I had been in jail and I didn't say anything, but, of course, I had been in jail."

Rittenberg did not describe the prison, other than to say the guards had been friendly. "Each of the guards would try to help me," he said. "They'd each bring me special messages, such as that my wife and children were all right, and yet none of them knew that the others were trying to help me too."

While working as an interpreter and editor he had access to several foreign and American newspapers and magazines, but in jail, "I was lucky to get the People's Daily." He asked what journalists in Hong Kong thought of the official New China News Agency dispatches, which Rittenberg apparently has something to do with now. "I'm trying to get them to do stuff that the Western press can used directly. All that stuff about the judicial system I didn't think was too good," he said, referring to Peking's recent endless, detailed explanations of its new constitution and legal system.

Rittenberg's Chinese guide, who had been elsewhere on the boat, suddenly appeared and broke up the conversation. "This is a scenic spot, a very scenic spot. You must not waste a chance to see it," he said, motioning Rittenberg to a good vantage point. The American smiled and agreed.

Rittenberg appeared to have no one from Peking accompanying him other than his wife, which indicated no great official concern over his movements. His guide here was not even from a provincial level office, but was merely a functionary of the local city foreign affairs office. The Chinese usually screened off Rittenberg's table in the hotel dining room and his guide seemed nervous about chats he had with American tourist, particularly two known to be journalists. Yet Rittenberg remained talkative and friendly when approached at breakfast the day after the boat ride, and later when he was asked to pose for a picture.

"I haven't talked to the press before," he said, "but I sort of like to play things by ear."

Unna, who met Rittenberg at an Army language training program at Stanford in 1943, recalled the young southerner talking about his distinguished political family. His grandfather was speaker of the state legislature, his father an acting mayor of Charleston. After attending the University of North Carolina he broke away to help organize tobacco-plantation workers and public service workers. At the Stanford program, Unna said, "He was absolutely the most brilliant linguist I had ever run into." Rittenberg was also an admirer of socialism and made friends among many of the leftist professors on campus, but he told Unna he was not a member of the Communist Party.

Rittenberg's first wife divorced him before the signal corps group left for duty in India, then China. Rittenberg, Unna said, was an unlikely soldier with dirty fingernails and a uniform that looked like it had been slept in. But he loved China, and when the war ended he went to Shanghai to work for an international relief organization.

There Rittenberg met some communists. "After a while they asked me if I wanted to stay and work with them," he said. "I wanted to write, so I said yes." One Red Guard document said he traveled to Yenan, the wartime communist headquarters, and was briefly arrested, then cleared by high officials who later fell in the Cultural Revolution.

That part of his life, however, remains murky. Unna cannot remember Rittenberg's name even coming up in any of the congressional hearings of the McCarthy era, when many Americans with much more fleeting contacts with the Chinese communists were publicly scourged.

His influence grew gradually and quietly in the colony of foreigners working for the Chinese government until 1967, when Mao's Cultural Revolution pushed him into the headlines. As Red Guard groups organized themselves to respond to Mao's call to bombard the bureaucracy, many of Peking's foreigners also joined "rebel" groups in the foreign language institutes and foreign broadcast and publications offices where they worked.

Rittenberg helped overthrow the leadership of Radio Peking with the approval of a fast-rising radical publicist, Wang Li, who had become very close to Chiang Ching.

The American was appointed with two Chinese to a three-member ruling committee which, the Milton's said, consisted "of the brilliant Rittenberg who made policy, [a] not at all brilliant young man who put the policy in writing, and [a] very inexperienced young woman who read the decisions aloud to the assembled masses in her beautiful Peking accent. They became known as 'the brain, the scribe and the voice."

According to the Miltons, Rittenberg enthusiastically supported one celebrated rebel leader who had allegedly been committed to a mental hospital for criticizing Mao's archenemy, President Liu Shao-chi. Later it was sad the same man had also criticized Mao, however. One Peking office worker who later emigrated to Hong Kong remembered seeing Rittenberg address a Chinese rally - a remarkable honor for a foreigner - and critize then-foreign minister Chen Yi, a close Chou En-lai ally.

Foreigners like the Miltons had trouble understanding the byzantine shifts of Chinese politics in that tumultuos perid, and began to worry about being "used by political forces over which we had no control and about which we had little understanding." Yet, the Miltons said, Rittenberg seemed unconcerned. When they asked how he had ended up with such power at Peking Radio." He replied with his usual disarming frankness that he really didn't know."

By the fall of 1967 Wang Li and other fervent radicals had clashed with the army. Mao and Chiang Ching began to abandon them and take a more moderate line. Rittenberg's odd position as an American with great power in a Chinese government became a convenient focus for attack. At the time he was jailed one Red Guard group published an article about him entitled, "A Mysterious American."

"Who is Rittenberg?" asked the Red Guard publication, the Canton Cultural Revolution Bulletin. "Rittenberg was a counter-revolutionary doubel-dealer . . . He was allowed to take part in our party's organizational life.However, he retained his American citizenship, drew a high pay, and regularly visited the embassies of capitalist countries. He is a man of doubtful antecendents and one to be suspected.

"People cannot help asking: How could a bourgeois politican from the United States usurp the leadership of a Chinese radio station? . . . Rittenberg became the number one man of the Broadcasting Bureau of China. Even he himself must have found it hard to believe in his own meteoric rise.

"Seeing this state of affairs, the masses of the Broadcastin Bureau said with great indignation: 'Is there really no one else in China? A bourgeois yellow magazine in Rittenberg's home country - the United States - cried with pride: 'Radio Peking is led by an American.' What cannot be tolerated if this can be?"

Two other foreigners experts, Israel Epstein and Michael Shapiro, were jailed on the same charges of helping organize an anti-Chou plot.

"The public charge was that we were American espionage agents," Rittenberg said at breakfast. "This is just speculation on my part, but I think this was really a way to get at Chou, since he had sort of sponsored us."

In 1973 Shapiro and Epstein were released and Chou met with them and other foreign experts to apologize for their imprisonment. He indicated they had been duped by Rittenberg, whom Chou maintained was a "very bad person." It is possible that Chiang Ching insisted Rittenberg remain in jail as a scapegoat for what her faction had actually been trying to do to Chou, and that Chou went along to preserve the shaky unity of the leadership.

Rittenberg has no complete explanation for what happened. Chiang and the other three members of her Gang of Four were all at the 1973 meeting with the foreign experts, listening carefully to what Chou said.

"Chiang Ching was the one who was talking about this," Rittenberg said. "In those days, if she said one sentence, that was enough, you didn't need any more evidence."

Rittenberg's returning to the United States to live or visit, as the Miltons and many other Americans who lived in China have done. But he said he is not sure he will. He is full of enthusiasm for his work and for the new Chinese life and policies that have emerged in the last two years.

His second wife, a gracious woman said to have been a student when they met, and their four children, aged 12 to 19. are free of the political cloud his imprisonment had put over them. They are living in the Friendship Hotel in Peking, waiting for better quarters to be arranged. Rittenberg prefers to minimize what happened to him and look ahead.

"We were just a small part of the witch hunt," he said. "We weren't the main thing.

"But I think what this has done is inoculate the country against witch hunts in the future. People are far more careful now about such things."