Egypt put its military on alert today, closing major highways from Cairo to the Libyan border and reportedly mobilizing troops amid suspicions that Libya was implicated in the hijacking of an Egyptian airliner that ended tonight in Malta with a bloody assault by Egyptian commandos.

Western diplomats interpreted Egypt's mobilization more as a warning to Libya than as a possible preparation for war in an increasingly tense confrontation with Libyan leader Muammar Qaddafi that has been building for months.

Late into the evening, top Egyptian military spokesmen denied that anything unusual was happening, asserting that highways were open even where military roadblocks and miles of backed-up traffic were plainly visible.

Despite the high stakes, as the crisis unfolded, Egyptian President Hosni Mubarak went about his day with an air of confident calm. Having met with his Cabinet early in the morning to discuss the crisis, he apparently decided early on his course of action, then went out for several hours on a previously scheduled tour of towns in the Nile Delta. The capital and its officials conspicuously pursued business as usual.

Yet however soft the rattling of the saber, Egypt showed with its assault on the Boeing 737 that it was determined to use it against terrorists even at great risk, and officials and diplomats here suggested that broader action could still be contemplated if Libyan complicity in the hijacking is established.

"It's clear Egypt thinks Libya is involved in this," one western diplomat said in the hours before the rescue operation.

Tensions between Qaddafi and his neighbors have degenerated from a chronic problem to an acute one since the summer.

There also have been reports out of Washington that the Reagan administration has approved covert support for efforts aimed at provoking and possibly toppling the ruler so often linked to international terrorism.

But officials here cautioned that events on the runway in Malta are obscured by the violence with which the hijacking ended. Because the hijackers were killed, their connections may be hard to confirm, and the ultimate impact of the affair is likely to emerge only as evidence is sorted out in its aftermath.

Mubarak "is not given to adventurism," one well-informed diplomat said.

But referring to the military alert here, he said that "if you were handling this crisis, you would do your utmost to keep Qaddafi off balance," including implied threats to use the full force of Egypt's Army.

An Egyptian source with close ties to the military leadership here described the commando attack as "a very successful operation," suggesting before the full extent of the bloodshed was apparent that in itself the assault might be sufficient vindication for Egypt.

But this source added, "Of course if Libya is involved, Mubarak is not the kind of man to joke around. He will be after them."

Because the hijackers made no detailed political demands, clues to their true motives and identity are few.

Egypt has seen development of subversive cells in the past both on the left and the Islamic fundamentalist right. Early reports said the hijackers called themselves representatives of a little-known group called Egypt's Revolution.

A group using the same name claimed responsibility for the slaying of an Israeli diplomat here in August and an attempt against another in 1984. After the August killing, a written statement by the group slipped under the door of a western news agency praised the Moslem militant who assassinated Anwar Sadat in 1981 and claimed that its goal was to make "the Israeli enemy" leave Egypt.

Throughout the day today Egyptian officials were anxious to assert that the hijackers probably were not Egyptians. "They are using the name to try to show there is no stability in Egypt," one military source said shortly after the plane was stormed.

Foreign Minister Esmat Abdel-Meguid told reporters here today that Egypt could not identify precisely the nationality of the hijackers, but he said he doubted that Egypt's Revolution existed as an organization. Abdel-Meguid and other Egyptian officials suggested that like Islamic Jihad in Lebanon, the name is probably a cover for groups with other identities.

There was considerable speculation here today that the hijackers were dissident Palestinians attempting to impede current Middle East peace efforts being pursued by Mubarak, Jordan's King Hussein and Palestine Liberation Organization Chairman Yasser Arafat.

Some Arab officials in Malta suggested that the hijackers had Palestinian accents.

The PLO strongly denounced the hijacking from the start.

But if the hijackers were dissident Palestinians, this would not rule out their having close ties to Qaddafi.

The Palestinian terrorist Sabri Banna, also known as Abu Nidal, who killed several moderate PLO officials in the past, recently was interviewed in Libya by western journalists and now appears to be based there.

Libya also maintains close ties to Iran, and such groups as the Iranian-backed Al Dawa party often linked to the region's widespread terrorism.

According to statements by U.S. officials in Washington and informed sources here, 26 transit passengers from Libya were on board the plane when it left Athens. Of these, 19 were carrying Egyptian passports, and the remaining seven were traveling on documents from other nations, which were not specified by these sources.

The history of bitter relations between Qaddafi and Egypt's leaders goes back more than a decade, and confrontation between Cairo and Tripoli has become a familiar feature of the region's troubled politics.

In 1977 the two countries fought a brief border war, then Egypt's peace with Israel provoked harsh attacks by Qaddafi.

During the past year, the situation worsened again as Qaddafi challenged Egypt for influence in Sudan after that country's April revolution, then expelled thousands of Egyptian and Tunisian workers from Libya.

But Mubarak appeared content to dismiss Qaddafi publicly as a lunatic and humiliate him by repeatedly foiling his efforts to assassinate Libyan dissidents and, last summer, an alleged attempt to blow up the U.S. Embassy.

The most recent alleged assassination plot was uncovered and foiled, with extensive and sometimes ludicrous television coverage, at the beginning of this month.

Egyptian officials said privately at the time that they expected Qaddafi to attempt some kind of retaliation, and that they were ready for it. But they warned that if he killed Egyptians in one of his operations, Qaddafi would have to face the possibility of war.

With a population of about 50 million, almost half a million men under arms and an arsenal of some of the most sophisticated U.S. weapons, Egypt generally is considered much more than a match for Qaddafi's nation of 3 million people and about 70,000 Soviet-armed soldiers.

But few analysts believe that Mubarak, whose main interest is his economy, would take any drastic steps unless sorely provoked.

As the smoke begins to clear from the runway in Malta, officials here still are waiting to see how far Qaddafi can be held responsible for the grisly flight of Egyptair 648.