Algeria sought today to underline its role as a trusted intermediary in the hijacking of the TWA airliner by insisting that it would have resisted any attempt by U.S. commando units to storm the plane during its two stopovers here.

The Algerian position on the hijacking, which was expressed in an editorial in the official newspaper El Moudjahid, reflected the delicate foreign policy balancing act of a post-revolutionary state that has tried to improve its relations with the United States gradually without cutting its ties to the radical Arab world.

Today's editorial in El Moudjahid said that the Algerian authorities had succeeded in securing the release of 86 passengers by gaining the trust of both the hijackers and the U.S. government. In return for releasing some of the passengers, the Algerians offered the hijackers a reasonably secure haven as well as publicity for their demands.

Commenting on news reports that a specially trained U.S. anti-hijacking force had been dispatched to the Mediterranean, the paper added: "Those who know us, particularly in Washington and throughout the Arab world, understand very well that the Algerian authorities would never authorize a foreign military force of whatever kind to operate on Algerian soil. And also that Algeria has the means to oppose it, if necessary."

When the TWA plane landed at Algiers, the Algerian authorities appeared to go out of their way to reassure the hijackers that force would not be used against them. The plane was parked on an isolated part of the airfield that was lit up by spotlights at night, making a surprise assault difficult.

Both Algerian and U.S. official sources insisted that no serious consideration was giving to freeing the hostages by force while the hijacked plane was on the ground at Algiers airport. Instead, the Algerian authorities tried to persuade the hijackers to exchange their captives for about 700 Shiite prisoners being held in Israel, according to a source involved in the talks.

Early on Sunday morning, according to the source, the negotiations appeared to be on the verge of success, with the hijackers dropping earlier demands for the release of Arab prisoners everywhere. International Committee of the Red Cross delegates, who were assisting in the talks, seemed to think they had Israeli approval for the release of the Shiite prisoners, and the hijackers seemed ready to accept the deal. The source said that the hijackers gave the impression that they might have relatives among the Shiite prisoners.

Algerian and Red Cross negotiators were unable to establish the exact political affiliation of the hijackers, or their relationship to Shiite extremist groups such as Islamic Jihad.

According to an informed source, the protracted negotiations broke down Sunday when the hijackers unexpectedly decided to fly the plane back to Beirut a third time without waiting for their own deadline to expire. Why the tacit agreement collapsed is still unclear. As the TWA airliner left Algiers, the spokesman for the hijackers told the control tower that they were ready to come back if the Red Cross was able to guarantee that all their demands would be met.

One participant in the talks speculated that there could have been disagreement over tactics between the original two hijackers who took over the plane in Athens and their Shiite accomplices who joined them during the second stopover in Beirut on Saturday.

Any attempt by U.S. special forces to seize the plane without the cooperation of the Algerian authorities -- on the pattern of the Israeli raid on Entebbe airport in 1976 -- would have been exceptionally hazardous as well as politically unacceptable. The relatively high degree of Algerian military readiness, as well as concern for the future of U.S.-Algerian relations, was probably sufficient to rule out such an attempt.

A source close to the negotiations with the hijackers said that the Algerian team was anxious to avoid any dramatic incident on Algerian territory, such as the killing of any hostages or the blowing up of the airliner.

Western diplomats here believe that the Algerian government could exploit its skillful handling of the hijack, and success in securing the release of a majority of the passengers, to win points in Washington. Until now, U.S. policy toward northwestern Africa has tended to revolve around Morocco, Algeria's western neighbor and historic rival.

Algeria played a similar intermediary role in the 1979-81 American hostage crisis in Iran. Algerian officials have expressed frustration that their contribution to securing the release of the hostages in Tehran was not publicly recognized at the time by the incoming Reagan administration, which tried to distance itself from President Carter's handling of the crisis.

It was not until Algerian President Chadli Bendjedid paid an official visit to the United States last April that the U.S. government formally put on record its gratitude to Algeria for acting as a go-between with the Iranians.

This time, by contrast, the Reagan administration has been quick to thank Algeria for its handling of the hijack crisis. A personal message from President Reagan was handed over to the Algerian leader yesterday by the U.S. ambassador in Algiers, Michael H. Newlin.