At a time when Libya's Muammar Qaddafi was dismissing U.S. charges that he has ordered the assassination of President Reagan, the hijacking of a Libyan airliner that ended tonight with the release of the 35 hostages unharmed at Beirut International Airport rattled embarrassing political skeltons in his closet.

The three hijackers' demands for the return of missing Shiite Imam Mousa Sadr, who his supporters charge was kidnaped in 1978 on Qaddafi's orders, has reopened one of the most awkward chapters in Qaddafi's effort to play in the dangerous game of Middle East politics.

The hijackers took over the Libyan jet Monday on a flight from Zurich to Tripoli and hopped around between Beirut, Rome, Athens, and Tehran. They flew here today -- the third landing in Beirut since the hijacking began -- and surrendered to Syrian peace-keeping troops.

The hijackers certainly must have known that their action was unlikely to produce their revered religious leader, who was last seen in Tripoli in August 1978, reportedly following a heated row with Qaddafi.

The bearded, Iranian-born imam's picture still dots the walls and lampposts of crowded Shiite neighborhoods in southern Beirut, and he is still invoked as the leader of Lebanon's 900,000-strong Shiite community. But black flags that fly from the balconies of many of his followers' apartments attest to the general belief today that Sadr is dead.

Qaddafi has vehemently denied any responsibility for the imam's disappearance, insisting that Sadr and his companions had boarded their flight for Rome and left Libya. Italian police have established that the imam's luggage arrived in Italy, but the lack of any evidence that would show that Sadr himself had arrived has led them to conclude he was not on the flight.

The Italian findings have underscored reports in Beirut that Sadr had been intercepted by two jeeploads of Libyan intelligence agents, taken into the Libyan desert and shot.

Whatever the case, Sadr's disappearance represented a major political stumbling block for Qaddafi. It turned the Lebanese Shiites, the country's largest single religious community, against him and brought against him the vengeful opposition of the Shiite Amal (Hope) militia, to which it is understood the three hijackers belonged.

It has also caused Qaddafi no end of embarrassment in his relations with Tehran's Shiite Islamic republic, although the Iranians' beleaguered leaders have since dropped the matter out of the need for Qaddafi's support in the Arab world.

Behind the story of Sadr's disappearance lies a bizarre story of Qaddafi's buying into the complex game of violence and influence-peddling that constituted Lebanese politics during the 1975-1976 civil war, hoping to match the more experienced traditional players from Syria, Iraq, Egypt, Israel and numerous foreign intelligence services.

His ante, according to knowledgeable Arab analysts, was a $10 million donation to Imam Mousa Sadr. This formed the basis of the Shiite leaders' funding and arming of the Amal militia. The condition for Qaddafi's donation, according to these analysts, was that Sadr's militia would ally itself with the Palestine Liberation Organization, which Qaddafi most desperately sought to impress and influence.

Qaddafi, however, failed to understand the Shiites' deep and bitter grudges against the PLO. Since 1970, the PLO had concentrated its forces in the mountainous south of Lebanon near the Israeli border. Prior to the PLO's arrival, the area had been mostly one of Shiite villages, and the often arrogant PLO presence there, in addition to Israeli bombings and ground attacks, created much bad blood between the two.

When in 1976 Syrian troops first entered Lebanon to quell the civil war that was raging out of control, it was the PLO, not the anti-Moslem Christians of the north with which Syrian forces first clashed. Sadr's militia remained neutral, much to the rage of Qaddafi, and eventually aligned with the Syrians.

The alleged row with Sadr in Tripoli in 1978 reportedly was over Qaddafi's demands for return of his $10 million, which by then long since had been spent.