The Palestine Liberation Organization's mediation mission to Tehran is showing signs of exacerbating tensions that have been mounting recently within the guerilla group.

The strains have arisen as Palestine guerilla leader Yassaf Arafat has tried to balance international diplomatic gains with a need to maintain revolutionary credentials.

As a result, the latest initiative in Arafat's campaign for respectability -- a tentative mediation effort to obtain the release, or at least ensure the safety, of American hostages being held by Iranian militants in Tehran -- has become fogged by confusing signals and contradictory statements from various PLO officials.

The main competition within the PLO is among forces supporting Arafat's new diplomatic approach, designed to win favor in Washington and European capitals, and other factions for whom Arab revolution and armed struggle remain the only true path to Palestine.

Despite Arafat's new emphasis on PLO diplomacy, many Palestinian leaders are still wed to the revolutionary principles with which the PLO rallied its supporters after the 1967 Middle East war. Arafat is obliged to pay his respects to these followers and preserve their loyalty even while he seeks new allies in the West.

It is this need to play to several audiences at once that helps account for the seemingly contradictory shifts in PLO policies as Arafat maneuvers from one declaration to the next toward a line that depends more on persuasion than violence.

Rarely has this been more apparent than in the PLO's handling this week of the Tehran mediation issue.

A PLO representative in New York, reportedly after conferring with Arafat, first announced that the guerilla group would mediate to prevent bloodshed in the embassy takeover. But in Beirut, at the PLO headquarters, guerrilla officials professed to know nothing about any such mediation.

When Arafat dispatched some PLO officials to the Iranian capital, his spokesman switched to saying they had orders only to make themselves available to help in any way they could.

A top Arafat advisor and the head of the PLO's office in Tehran, Hanai Hassan, said even as he departed Beirut to help run the PLO team, "The Palestine Liberation Organization cannot mediate between Iran and the United States because the PLO is on Iran's side."

Another example of the pressure Arafat faces in his drive to portray the PLO as a moderate force came from Issam Qadi, the new head of the pro-Syrian Saiqa guerrilla group. Speaking in Damascus, Qadi said the PLO should support Iran's revolution outright and could not "offer its services to American imperialism as a way of trying to get recognition from the U.S."

The PLO's equivocal approach was illustrated again yesterday by the revelation from high Lebanese sources that Palestinian forces early this week rounded up 40 Iranians who had planned to capture the U.S. Embassy here in an operation similar to the Tehran takeover.

Officially, PLO spokesman said they had no information on the report. Privately, they expressed satisfaction today that such news was out. Arafat himself was said to be pleased that a PLO "good deed" for the Americans had become public without passing through the PLO propaganda mill.

Whatever the exact truth of the reported embassy takeover plot, the PLO once again had been depicted as a moderate force willing to use its influence and arms in a respectable way. This was seen as another boost for Arafat's campaign to win recognition as a valid partner for the United States in talks on the Middle East.

Again illustrating the delicate balance Arafat has had to keep, however, PLO officials telephoned and prevailed on some pro-Palestinian newspapers in Beirut last night to skip the story because it might have been poorly received locally.

Most Palestinians and their Lebanese supporters identify with Iran's Islamic revolution. Some PLO offices display posters glorifying Iranian Moslem triumphs over the shah and depicting the shah's downfall as another milestone on a revolutionary road leading to victory over Israel.

A Palestinian intellectual with close ties to the PLO explained that Iran's revolution holds great appeal for many in the guerrilla movement because, in their view, the PLO stands for social revolution as well as the fight to regain Palestine.

Most PLO members come from lower middle-class families, he said, and tend to blame the loss of Palestine to Israel on treachery by wealthy Palestinian landowners as well as Israeli military victory. In this perspective, the triumph of lower class Iranian Moslems over the shah's monarchy provides an attractive model for what the young guerrillas hope will be their own victory some day.

In addition, he said, the PLO leadership itself got its start in an atmosphere created by Algeria's victory over French colonialists, the triumph of Fidel Castro in Cuba and the Vietnam war, where American firepower was being frustrated by the Viet Cong.