Islamic nations embarked on a limited mission to end the war between Iraq and Iran yesterday, but the Iranian leadership immediately rejected any appeal for negotiations and said the Moslem envoys should first recognize Iraq's "aggression" against, Iranian territory.

Foreign Minister Agha Shahi of Pakistan, acting chairman of the 40-nation Islamic Conference, announced at United Nations headquarters in New York that the Islamic "goodwill mission" will include President Mohammed Zia ul-Haq of Pakistan and Habib Chatti, the Tunisian secretary general of the Islamic Conference.

Given Iran's reluctance and their limited mandate -- restricted to preliminary contacts in Tehran and Baghdad, specificially excluding mediation given little chance of arranging an early end to the five-day-old Persian Gulf conflict.

This assessment was reinforced by uncompromising remarks from President Abol Hassan Bani-Sadr of Iran broadcast over Tehran radio as well as a Foreign Ministry statement in Baghdad reiterating that Iraqi conditions for a halt in the fighting include recognition of complete Iraqi sovereignty over the strategic Shatt-al-Arab estuary at the head of the Persian Gulf.

"We are being attacked," Bani-Sadr said in meetings with the Pakistani and Algerian ambassadors in Tehran, according to the radio. "Therefore, to end this aggression, it is Iraq that will have to take steps. It is Iraq which will have to end its aggression . . . how can one mediate between good and evil?"

The Iranian prime minister, Mohammed Ali Rajai, rejected the Islamic peacemaking attempt even more categorically, stating on Tehran radio soon after Chatti's announcement:

"We will not accept any mission as a good-will delegation, and we will never be ready to speak one word about negotiations."

Also in the background were intense fighting for control of the besieged Iranian port of Khorramshahr and Abadan along the Shatt-al-Arab waterway and an Iraqi drive toward the key Iranian city of Dezful, controlling vital rail and road links between Khuzestan Province and the rest of Iran.

These battles were shaping up as decisive tests and neither Iraq nor Iran was expected to cooperate in the Islamic peace mission until events on the ground turned more clearly one way or the other, forcing one side to abandon or scale down its objectives.

Reports from Baghdad suggested that President Saddam Hussein of Iraq and his military commanders have been surprised by the strength of Iran's response to attacks along the border. It was only after failing in their original plan to seize both banks of the Shatt-al-Arab swiftly did the Iraqis decide to expand and prolong the conflict, the reports said.

At the same time, Saddam Hussein's drive for leadership in the world nonaligned movement and preeminence among Arab states around the Persian Gulf was seen as a likely encouragement for him to end the fighting as soon as possible. Occupation of foreign territory would be an embarrassment within the nonaligned movement, particularly since Baghdad is to host its next world conference. In addition, Iraq's Persian Gulf neighbors already are nervous lest the fighting endanger their oil routes.

The visit to Baghdad yesterday by Cuban Foreign Minister Isidoro Malmierca was seen as part of these concerns. After a meeting with Saddam Hussein, Malmierca conferred with Iraqi Foreign Minister Saadum Hammadi and the Iraqi declared:

"Iraq will not keep a silver of Iranian territory if Iran recognizes Iraq's sovereignty over its land and the waters of the Shatt-al-Arab, refrains from interfering in Iraq's internal affairs and respects the independence of neighboring countries."

Diplomats at the United Nations said it was Iran's representative, Jamal Shemirani, who insisted that the Islamic mission be barred from mediation attempts.

"We are victims of naked aggression and there is nothing to mediate," they quoted him as telling a closed meeting of Islamic delegates that decided on the mission.

Iran has been reluctant to cooperate with the United Nations since the Security Council called last November for release of the American hostages held in Tehran by Iranian militants. In addition, Iran's objections were believed to reflect fears that any call for a cease-fire would aim at halting the battles with Iraq still controlling patches of Iranian territory that would give Baghdad strong bargaining leverage in any subsequent negotiations.

The Security Council moved into action for the first time tonight after waiting for the Islamic group to launch its initiatives. Norway and Mexico circulated a resolution calling on both sides to "refrain immediately from any further threat or use of force," to "settle their dispute by peaceful means," and to "accept any appropriate offer of mediation" -- a reference to the Islamic mission.

After brief speeches by the resolution's sponsors, Iraqi representative Ismit Kittani and Secretary General Kurt Waldheim, the council adjourned until this afternoon.