An anonymous caller claimed today that five American hostages had been killed by their captors, but no bodies were found in the abandoned factory where the caller said the bodies had been left.

Two calls to a foreign news agency said the shadowy group Islamic Jihad, whose name means holy war, had executed the six hostages "by firing squad" and discarded their bodies underneath a devastated soft drink factory near the Cola Roundabout in Moslem-controlled west Beirut.

A thorough inspection of the rubble-strewn basement of the factory, underground wells and tunnels by police and reporters showed no trace of the missing Americans. The authenticity of Islamic Jihad communiques to international and local news organizations is impossible to verify.

The calls followed a report by Assem Kansou, the head of the pro-Syrian Lebanese Baath Party, that the radical Hezbollah, or Party of God, had been given an ultimatum to evacuate a Lebanese Army barracks near Baalbek that it had commandeered in 1983. The Syrian Army controls the Bekaa region in central Lebanon, where Baalbek is located.

Hezbollah, which may or may not be structurally linked to Islamid Jihad, espouses the same religious goals and seems to have similar political views and sympathies with Iran.

The ultimatum to Hezbollah coincided with Syrian efforts to promote a national reconciliation and agreement on constitutional reforms. The radical Shiite group, left out of the bargaining process involving Lebanon's main warring factions, has expressed its opposition to such an accord.

Islamic Jihad, which claims to be holding at least five Americans kidnaped in west Beirut since March 1984, has offered to release U.S. and French hostages in return for the freedom of 17 prisoners held in Kuwaiti jails on charges of carrying out a series of bombing attacks on Dec. 12, l983.

On Oct. 4, Islamic Jihad announced that U.S. Embassy political officer William Buckley, who was kidnaped in March of last year, would be executed in retaliation for an Israeli air raid against Palestine Liberation Organization headquarters in Tunis. Buckley's body has not been found.

Islamic Jihad has claimed responsibility for the abduction of Buckley, the Rev. Lawrence Jenco, head of Catholic Relief Services in Lebanon; Associated Press bureau chief Terry Anderson; David Jacobsen, director of the American University of Beirut hospital, and Thomas Sutherland, dean of the university's agriculture school. The university's librarian, Peter Kilburn, was also reported missing last December.

Three Soviet Embassy employes who had been kidnaped by the Islamic Liberation Organization were released Oct. 30 following intense efforts by leftist groups and Syrian intelligence officers to locate them. A fourth Soviet hostage was killed.