Egypt, frustrated in its attempts to come up with proof that Libya mined the Red Sea last summer, appears ready now to make the most of its successful "sting" operation that thwarted a Libyan assassination attempt and led to a false and embarrassing announcement by Tripoli that it had succeeded.

Egyptian President Hosni Mubarak denounced Libyan leader Muammar Qaddafi today as "an international terrorist" and urged other countries to take steps to "stop him and make him realize his limitations."

Egypt, according to observers, intends to use the trial of the four would-be assassins of a Libyan exile here as a forum for symbolically convicting Qaddafi in absentia in the international limelight The arrest of the four men, two Britons and two Maltese, was announced yesterday.

The Egyptians are so pleased with their antiterrorist operation -- one of the Arab world's most successful in years, with doctored photos and phony messages -- that they are now planning to make a documentary film of it to show around the world.

They boast that not only did they foil the plot to assassinate Abdul Hamid Bakoush, a former Libyan prime minister and a strong foe of Qaddafi, but they managed to infiltrate and manipulate the Libyan intelligence services into making a fool out of Qaddafi.

Qaddafi, however, with his Bedouin pride, is considered unlikely to take the turning of the tables on him lightly, and the expectation here is that he will try to take revenge in his own time and manner.

Qaddafi, at a rally in the Mediterranean island nation of Malta, where he is on an official visit, ignored the Egyptian allegations in a speech. He did declare that "the president of the United States is mad, mad." In Tripoli, the official Libyan news agency Jana denied the Egyptian charges, which it said were a prelude to aggression against Libya, Reuter reported.

Libya's number two leader, Abdelsalam Jalloud, was quoted by Jana, however, as saying Egypt would not be able to protect Bakoush "even if Hosni Mubarak put all the Egyptian Army" to that purpose, The Associated Press reported.

Egypt also is expected to use the publicity of a trial to try to block Qaddafi's current efforts to improve his relations with West European leaders, notably President Francois Mitterrand of France and Chancellor Helmut Kohl of West Germany.

The Reagan administration has attacked Qaddafi as a chief fomenter of international terrorism and is expected to welcome this windfall in its campaign to isolate him from the world community.

Mubarak, talking to reporters today, gave a preview of Egypt's apparent campaign to expose what it sees as Qaddafi's international terrorist machinations.

He accused Qaddafi of planning to assassinate a wide array of world figures, including Mitterrand, Kohl, British Prime Minister Margaret Thatcher, and the leaders of Pakistan, Saudi Arabia, Kuwait and the United Arab Emirates. He accused Qaddafi of involvement in the assassination last month of Indian prime minister Indira Gandhi. He offered no proof for his allegations, saying only that Qaddafi had done it "by financing some organizations to commit all these crimes."

The extent to which Egypt's security and intelligence services were able to infiltrate the Libyan ring of would-be assassins became clear last night in a press conference given by Egyptian Interior Minister Ahmed Rushdi.

Rushdi provided reporters with a wealth of names, dates, copies of letters, pictures, telephone numbers and day-by-day information on how the Libyans tried to sneak a hit team into Cairo to "execute" Bakoush, 46, leader of an anti- Qaddafi group based here.

As it turned out, Qaddafi's "suicide squad," as it was described in the Libyan news agency's erroneous announcement, consisted of two apparently naive middle-aged Englishmen and two Maltese who were led around by the nose from the moment they set foot on Egyptian soil, then set up, used and arrested.

Rushdi said Egyptian intelligence had learned of the plot in advance and placed undercover agents inside the Libyan ring. It was they who apparently were supposed to carry out the assassination.

One of the Egyptian agents tricked the head of the Libyan squad into writing a letter Nov. 12 to the head of Libya's embassy in Malta saying that the "execution" had been carried out and asking for the remaining $90,000 out of the $150,000 they were to be paid upon completion of the job, Rushdi said.

Another Egyptian agent took the letter to Malta, where, Rushdi said, he gave it to a Libyan agent and then vacationed there at Libyan expense.

On the basis of this letter, Ali Nagem, head of Libya's embassy in Malta, sent a message to Ahmed Toumi, a top intelligence aide in Tripoli, telling him Bakoush had been killed, Rushdi said, and Toumi then messaged the assassination squad in Cairo expressing the Libyan leadership's "happiness" over its success.

Rushdi said the money was then sent from an unnamed bank in Rome to the American Express Bank here, and Nagem went to the Greek island of Crete with the "news" of Bakoush's assassination and the pictures as proof, to tell Qaddafi, who was meeting there with Mitterrand.

Qaddafi then told Jana to announce the presumed successful "execution" of Bakoush, according to Rushdi. Jana and Tripoli radio made the announcement Friday, saying that the "stray dog" Bakoush had been "executed" on Monday by a Libyan "suicide squad."

Rushdi said that Anthony William Gill, 48, a Briton and one of the four men arrested, "confessed that he met with the international terrorist Carlos more than once at a hotel in Tripoli." Carlos, whose real name is Ilich Ramirez Sanchez, is believed to have masterminded several international terrorist attacks and airplane hijackings.

Rushdi said the Libyan agents made 16 phone calls to Tripoli and London during preparations for the assassination, and he showed reporters the chits for the calls from the Egyptian telephone company.

Why the agents thought they could make phone calls to a Libyan intelligence office in Tripoli from Cairo without fear of being tapped, given the present hostile state of relations between the two countries, remains unclear.

Apparently, none of the four plotters ever actually saw Bakoush. He had been taken by Egyptian agents to the safety of Aswan in Upper Egypt, where the phony picture of his "execution" was taken.

A potential flaw in the "sting" operation went unnoticed by the Libyans.

Pictures of the "dead" Bakoush showed him sprawled on his back, apparently shot in the forehead. But the blood, instead of flowing from the wound onto the floor as gravity would have it, is shown splattered down his face and the front of his shirt.

For this to happen, he would seemingly have had to be kept standing upright after being shot. But this detail apparently went unnoticed.