Prime Minister Mohammed Ali Rajai of Iran tonight firmly refused to meet U.S. government officials seeking release of the American hostages, but he discussed the captives' fate with U.N. Secretary General Kurt Waldheim and Islamic diplomats.

"No, never," Rajai told a reporter who asked whether he would respond to the Carter administration's proclaimed willingness to meet him and discuss U.S.-Iranian differences and the future of the 52 hostages.

Rajai's categorical refusal at a news conference cast a shadow over hopes here and in Washington that his presence in the United States, if even for two days, could provide an opening for high-level direct contact between the United States and Iran over the hostages.

Earlier, addressing the U.N. Security Council, Rajai rejected Islamic appeals for a cease-fire in the Iranian-Iraqi war and accused the United States of helping Iraq in an effort to force the hostages' release.

Donald McHenry, U.S. ambassador to the United Nations, said after Rajai's declarations, his meeting with Waldheim and address to the Security Council that the net result of Rajai's appearance here was "no movement" in American efforts to obtain the release of the hostages.

U.S. diplomats indicated, however, that they still hope for progress from Rajai's discussions with Waldheim and the Islamic representatives here, all of whom have been briefed by U.S. officials and have pledged to urge Rajai to arrange the hostages' release.

McHenry said that the Islamic representatives' conversations with Rajai tonight were designed to impress upon the Iranians that Iran is isolated in its insistence on retaining the hostages and that in the long run the best course to obtain international backing in its war with Iraq is to release them.

McHenry said Rajai's condemnation of Iraq was ironic because the Iranian premier charged that Iraq had violated human and legal principles by attacking Iran. Iran itself was violating these principles by holding the Americans hostage, McHenry said.

"These principles of international law are principles for everyone," he said.

At the same time, McHenry said that Rajai's very presence in the United States was a "first step." The Iranian prime minister, a 46-year-old former schoolteacher with little government experience, is the highest ranking Iranian to set foot in the United States since the American hostages were captured in the U.S. Embassy in Tehran on Nov. 4, 1979.

"But whether we're going to get beyond that first step, I just don't know," McHenry added.

At his news conference, Rajai showed reluctance to discuss the hostage issue, saying he would respond to questions on the hostages Saturday. This set off speculation that he wanted to leave the way open for further exploration of ideas raised in his meeting with Waldheim or by Islamic authorities. It was also evident that Rajai wants to focus attention on his country's grievances against Iraq in their war, now in its 26th day.

Rajai rejected new appeals by Islamic nations for a cease-fire for the Moslem holiday of Aid al-Adha, calling the offer "another deceiving act."

At the Security Council, where he delivered a 70-minute speech, Rajai stuck to his line as he declared that "a fair settlement can be found for this war only if the aggressor is conquered and punished."

In linking the United States with the Iraqi war effort, the Iranian prime minister charged that intelligence obtained by U.S. Air Force AWACS radar command planes -- recently delivered to Saudi Arabia -- was being supplied to the Iraqis.

To Rajai's dismay, the hostage issue overshadowed the Persian Gulf war he said he had come to discuss. In his Security Council speech, Rajai depicted Iraq as an aggressor cooperating with "the superpowers" to crush Iran's Islamic revolution and compared the Iraqi government of President Saddam Hussein to colonialists, Zionists and imperialists.

"From this rostrum, we would declare to the whole world that Iraq's aggression is an integral part of the international aggression directed against our revolution," he said, reading in a slow, measured voice from a prepared text. "We would repeat, Iraq's aggression grows from the forces of evil, and it plays into the hand of the great satan, and the lesser satans as well." Rajai was referring to the United States and its allies.

Rajai's Security Council address ended an Iranian boycott prompted by that body's resolutions calling on Iran to release the American hostages.

U.S. diplomats today repeated earlier American assertions that President Carter, Secretary of State Edmund Muskie or other American officials were ready to meet Rajai if he wanted to, but insisted the United States had taken no direct initiative to set up such a meeting.

"Our concerns are well-known," said Jill Schuker, spokeswoman for the U.S. Mission at the United Nations.

Rajai, wearing an open-necked yellow polo shirt and looking unshaven, met for 90 minutes with Waldheim before his Security Council speech. Assistant Secretary of State Warren Christopher saw Waldheim for 45 minutes yesterday in what U.S. officials indicated was a briefing for today's meeting to convey U.S. wishes on the possibility of contacts with Rajai over the hostages as well as the war in the gulf.

Ali Shams Ardakani, Iran's U.N. delegate, insisted today that Rajai's mission centered on the hostilities with Iraq. But he avoided a categorical declaration that the hostages could not be discussed.

"The country's at war, and you keep on talking about the hostages," he told reporters pressing him on the hostage issue. "We are worried about the war in my country."

Ardakani was responding to a welter of speculation here and in Washington that the Iranian decision to dispatch Rajai to the United Nations could signal a willingness to accept discussion on the hostages, perhaps leading to their release and a lifting of the U.S.-sponsored embargo. The embargo has prevented resupply of the Iranian military except for limited shipments from North Korea, Libya and, reportedly, Syria. The Iranian Army and Air Force is largely U.S.-equipped and would benefit from shipment of U.S. parts being withheld by Washington because of the hostages.

Ardakani told reporters, "Spare parts are your issue, not our issue. Anybody who volunteers help, we get it."

Rajai arrived from Iran shortly after midnight on a visa issued by the United States for three days and tied directly to his U.N. business. It bars his moving outside a 25-mile radius of the United Nations without special permission or a change of visa granted by the State Department through the U.S. Mission here.

Schuker said Rajai's plane brought 22 persons, including crew members who remained at the airport and 11 other persons who came with him into New York City. They were staying at an elegant Manhattan townhouse that previously served as the residence of the Iranian ambassador to the United Nations during the reign of the late shah. Reporters waiting outside the townhouse said Rajai stayed inside and received no one before departing for the United Nations.

Although overshadowed by the hostage issue in U.S. eyes, the Persian Gulf war was the center of attention for foreign diplomats here. Pakistani Foreign Minister Agha Shahi said President Mohammed Zia ul-Hag had telephoned the leaders of Iran and Iraq in his capacity as chairman of the 40-nation Islamic Conference to appeal for a cease-fire during the four-day Moslem Aid al-Adha feast marking the end of the annual pilgrimage period.

A high-ranking Iraqi diplomat here had predicted before Rajai's speech that Saddam Hussein's government would accept the appeal provided Iran also accepts.