An administration official said yesterday that there is "no viable military option" in responding to the current hijacking of a U.S. airliner because of the difficulty in determining who was involved and selecting an appropriate target.

At the same time, White House spokesman Larry Speakes shied from describing the hijacking as state-sponsored terrorism, saying, "I don't think we are prepared to draw that conclusion."

The two comments underlined the agonizing uncertainties facing U.S. decision makers yesterday as they weighed the possible resort to military force against the Shiite hijackers of the Trans World Airlines plane or their presumed allies.

The question was whether such action would deter or simply escalate the use of violence against American interests in the Middle East.

The Israeli experience in Lebanon the past three years has shown that striking back can be counterproductive, increasing the determination of Shiite extremist groups to retaliate.

Furthermore, the fragmentation of the Shiite movement in Lebanon has become so severe, with countless small factions apparently operating semi-independently of each other, that it is far from clear that a U.S. strike against one group of extremists would serve any purpose other than stirring the wrath of others.

In addition, the United States must weigh the effects of any action on its Arab allies, particuarly the vulnerable oil-rich Arab kingdoms and sheikhdoms in the Persian Gulf, which stand to suffer the consequences of any American strike, as well as American interests in the region.

A reminder of this linkage came to the fore as recently as late May when the White House repeated its warning to Iran that the United States "will respond with whatever action we deem most appropriate against those responsible for state-supported terrorism" if American hostages being held by Shiites extremists in Lebanon were killed.

The administration was responding to news reports that it had drawn up options for bombing Iranian ports, the big oil terminal on Kharg Island in the gulf or the Iranian holy city of Qom in retaliation for any action taken against the hostages.

The speaker of Iran's Parliament, Ali Akbar Hashemi Rafsanjani, warned a few days later that any U.S. raid on Kharg or attempt to close Iranian ports would result in the closure of other Arab ports across the gulf.

This Iranian threat of escalated warfare against the other Arab oil producers has been one key factor in Iraq's calculations of how far it can go in its threats to destroy the Kharg Island oil terminal.

Most of the Arab oil installations and ports are less than 20 minutes flight time from Iranian air bases, and those in Kuwait are particularly vulnerable because of that country's meager air defenses. Kuwait has already been the target of Iranian air and Shiite terrorist attacks, including a recent attempt to assassinate the ruler, Sheikh Jabir Sabah, with a car bomb.

In addition, Iran has obtained from Libya Soviet-made Scud missiles with which it has been hitting the Iraqi capital of Baghdad and could easily hit Saudi or Kuwaiti oil facilities.

It is a widely held assumption in Washington that Iran and Syria are responsible for much of the anti-American terrorism sweeping the Middle East and that the Shiite extremists in Beirut are simply their operatives.

U.S. and other western intelligence sources, including Israel's, have said there is strong evidence of Syrian and Iranian involvement in past terrorist incidents, such as the bombing of the U.S. Marine and French barracks of the multinational peace-keeping force in Beirut in October 1983.

Today, however, the continuing fragmentation of Shiite extremist groups in Lebanon has made it less clear to what degree Iran or Syria may be behind the terrorists. In some instances, it appears that small groups are acting on their own.

The recent spate of kidnapings of U.S. citizens in Beirut is one case in point.

A shadowy group using the name Islamic Jihad, or Holy War, has taken responsibility for six of the seven kidnapings and demanded the release of 17 Shiite terrorists in jail in Kuwait.

An Arab source familiar with the secret negotiations to gain the release of the Americans said the talks have been frustrating because more than one group is holding the Americans and they could not agree among themselves.

The main motivation for taking the hostages, he said, was that those being held in Kuwait are relatives of the Beirut kidnapers.

The confusion over who belongs to what Shiite group was again demonstrated Saturday by the confession of the 21-year-old accomplice of the original two TWA hijackers, Ali Atwa, who was arrested in Athens.

He was quoted by Athens police as saying they all belonged to Islamic Jihad.

The same day, the other two released a statement from the TWA plane, then in Algiers, denying they had anything to do with that group.

"One day they are members of Hezbollah, another day Amal and another Islamic Jihad," said the Arab source, referring to the names of just three of the Shiite groups operating in Lebanon.

Even Syria, which now holds the dominant hand over Lebanese politics, apparently has limited influence. Syian President Hafez Assad recently said he had been asked by President Reagan to help gain the release of seven U.S. hostages being held by another group of Shiite extremists in Lebanon.

Assad said he had succeeded in contacting those responsible, but had failed to persuade them that such actions were against "the code of honor" of those combating U.S. influence in the Middle East.

In the ever-growing political chaos in Lebanon, particularly in Moslem-dominated West Beirut, it appears that a multitude of factions of Shiite extremists have sprung up, making it difficult for anyone, including even the mainstream Shiite Amal leaders, to deal with them and their demands.