TRIPOLI, LIBYA -- Libyan leader Moammar Gadhafi's dealings in an international underworld peopled by terrorists and guerrillas, mercenaries and spies have been a major liability in his dealing with the West. Now he would like to turn them into an asset.

Four years after U.S. warplanes bombed Gadhafi's house here for his alleged support of terrorism, the Libyan leader seems eager to play the powerful penitent who can persuade Palestinian terrorist Abu Nidal to free hostages and can mediate between hard-liners in the Arab world.

Gadhafi's drive for Western acceptance is fueled largely by a realization that he has lost the generous support -- including military advisers and arms -- that he once enjoyed from the Soviet Bloc.

To cash in on his links to the terrorist underworld, however, Gadhafi must move cautiously. Winning Western approval requires proof that he can exert a positive influence on terrorists such as Abu Nidal, without appearing to be their patron. But cooperation with the West could shatter the radical credentials he still covets and make him vulnerable to terrorist blackmail.

"Gadhafi is in a real 'Catch-22' situation," said one European official who has studied Libya for the past decade. "He needs the West, and his Arab neighbors, more than ever before {but} it's doubtful he'll manage to please everybody."

At home, Gadhafi has shown himself to be a master at playing one interest group against another and emerging on top. Internationally, however, such tactics have produced more burned bridges than real allies -- something Gadhafi seemed to realize only after the air raid in April 1986.

Yet last month, Gadhafi's bargaining chips began to pay off with the still unexplained release of three hostages -- two French and one Belgian -- who ostensibly had been held by Abu Nidal and were freed upon Gadhafi's intervention. For his help, Gadhafi received the "personal thanks" of French President Francois Mitterrand and his foreign minister's description of the mediation as a "noble and humanitarian gesture."

According to Libya specialists, Gadhafi's overtures toward the West have been encouraged by the withdrawal of East European military advisers and arms supplies that once poured into this country.

At a recent meeting here of Palestinian leaders, Gadhafi's deputy, Maj. Abdul Salaam Jalloud, accused the Soviet Union of abandoning its "policy of struggle." He also urged the Palestine Liberation Organization to renounce its moderate course and "push Israelis into the sea." But Jalloud then sat down, acknowledging that others thought differently, and listened quietly while Palestinians -- excluding Abu Nidal's faction -- criticized his stance, urging cool heads to deal with Israel.

Such a public debate would have been unthinkable even three years ago, when Libya was still solidly allied with radical partners Syria and Iran, and before Gadhafi grudgingly made peace with PLO leader Yasser Arafat. While the state of Israel remains unacceptable to Gadhafi, pragmatists within his regime have been mending fences with Egypt, despite that country's peace treaty and diplomatic relations with the Israelis.

"Until two years ago, Iran was a big friend here. Why? Because revolutionary considerations were predominant," one Western diplomat said. "Now Gadhafi is trying to mediate between Egypt and Syria, Syria and Iraq, Arafat and his Palestinian opponents. In 1986, the Palestinian orientation was Abu Nidal" and other radicals. "Now it's Arafat. Which is preferable for the West?"

Gadhafi, in an interview with an Egyptian newspaper last fall, admitted he had "erroneously" supported terrorist groups ultimately "harmful" to the Arab cause. He reportedly has cut back aid to a number of rebel or terrorist groups. And, in an apparent bid to win back British relations, he kept Abu Nidal under house arrest for several months near Tripoli, according to a broad range of sources.

But Gadhafi has not turned his back on revolutionary causes. Last fall, Jalloud boasted that 22 percent of Libya's income -- or more than $1 billion a year -- funds "liberation movements." Rebels who espouse "anti-imperialist" causes are still regular guests here. And recently two Libyan "diplomats" were expelled from Ethiopia for allegedly planting bombs aimed at an Israeli envoy.

Gadhafi's relationship with Abu Nidal apparently remains close -- at least if the hostage affair is taken at face value.

In Tripoli these days, few bother to deny any more that Abu Nidal, whose real name is Sabri Banna, has set up shop here. His aides seem to move freely, appearing at a downtown hotel early this month to pass out leaflets castigating Arafat, also in town at the time. One Abu Nidal aide, who would not give his name, said his leader was outside Libya but would not say where. Western analysts and other Palestinians said this was also their understanding.

Although his health has suffered -- most likely from a heart attack, according to a variety of sources -- and infighting among his followers last year left dozens dead, Abu Nidal is still deemed capable of launching a terrorist strike. Diplomats say he operates two or three training camps here, but PLO officials deny this. Yet even Palestinians who like to point out how many former Abu Nidal supporters now support Arafat say they fear more violence.

"He is weakened, but in that condition he may become more desperate to prove himself," one Palestinian source said. "We are all working to prevent something that would mean ruin for all of us."

Observers here say the United States is looking for evidence of changed behavior on Gadhafi's part, and they say a Libyan break in ties with Abu Nidal may be a key part of that. But because of their past relationship, Abu Nidal and Gadhafi are each thought likely to be vulnerable to blackmail by the other, and, analysts say, they may still find parts of their relationship useful. Thus, they say, it is unlikely that Gadhafi and Abu Nidal will break completely.