Iran's decision to resort to the use of force against the hijackers of the Kuwaiti airliner was believed here to have come only after an anguished debate within the Tehran government over how best to handle a crisis involving some of its closest allies abroad espousing a cause dear to its own heart.

The hijackers are believed to be members of an extremist Shiite Moslem group with strong ties to the Iranian rulers. Furthermore, they had been making demands on Kuwait that Iranian authorities themselves have long been voicing: the release of 17 convicted terrorists belonging to an Iranian-backed Iraqi opposition faction known as Al Dawa.

Because of the importance of the religious establishment in Iran and its dedication to the spread of the Iranian Islamic revolution abroad, the Tehran government found itself from the start in an awkward position in dealing with the hijackers.

On the other hand, at least one faction in the Iranian government, led by the Foreign Ministry, has been working hard for the past six months to improve Iran's image and relations with Western European countries and even the conservative Arab states of the Persian Gulf.

Kuwait, long suspicious of this new Iranian policy, was convinced almost from the start that Iranian authorities were actively aiding the hijackers. Earlier today, Kuwait had finally given up on dealing with Iran and ordered home the team they had sent to Tehran to help in the negotiations.

The decision to bring the three-man Foreign Ministry delegation back was viewed by western analysts here primarily as a move to signal Kuwait's extreme anger with the way Iran had been handling the hijacking and to stress Kuwait's contention that it was time Tehran intervened to end the crisis.

Kuwaiti sources said the Iranians had generally ignored Kuwait's delegation, even barring it from talking to two Kuwaitis taken earlier from the plane to a Tehran hospital for treatment. Since Kuwait had no intention of negotiating with the hijackers, there was no reason for the delegation to remain in Tehran, the sources said.

Nonetheless, coming after two days of charges and countercharges over who was responsible for the impasse, the action was seen here as one more sign of the sharp deterioration in relations between Kuwait and Iran over the hijacking.

Iran blamed Kuwait Saturday for its initial decision not to use force against the hijackers. But Kuwait said it had told Iran from the start it was leaving it totally up to Iran how to deal with the crisis and ensure the safety of the passengers.

Kuwait never formally accused Iran of complicity in the hijacking, but the press here, clearly reflecting official thinking, more or less did, pointing to the assistance Iran gave the hijackers in publicizing their statements.

Kuwait also noted the similarity of the demands the hijackers made with those over the past six months by Iran regarding the release of the 17 convicted terrorists here.

Kuwait also pointed out that Al Dawa, the Iraqi opposition group to which the hijackers are believed to belong, is based in Tehran and that its leader, Mohammed Bakr Hakim, has long been an official guest of Iran. Furthermore, Tehran has backed Al Dawa's terrorist activities against Iraq and Israeli and American targets in Lebanon.

It is not known whether the hijackers are members of Al Dawa or some similar allied Shiite extremist group. But they use the same kind of religious-oriented militant language and consistently referred to the convicted terrorists here as their "brothers."

Kuwait has indicated that it is convinced that Iran was responsible for the whole affair and is using it to embarrass and put pressure on the government here.

Iran may have been less embarrassed by these accusations in the Kuwaiti press, however, than the mounting international pressure on it to do something or assume responsibility for a disaster in the making.

Kuwait had mounted a major diplomatic campaign against Iran and even recruited Syria, Iran's closest Arab ally, whose support in its war against Iraq it values highly.

A Syrian delegation arrived today in Tehran, and Kuwait interpreted this as a positive sign that the diplomatic tide was running strongly against Tehran.