President Carter and Secretary of State Edmund S. Muskie offered yesterday to meet Iranian Prime Minister Mohammed Ali Rajai in an attempt to win the freedom of the 52 Americans held hostage in Iran for almost a year.

Rajai arrived in New York City early today to take part in a United Nations Security Council debate on the Iran-Iraq war. His unexpected trip immediately triggered a spate of rumors about a secret deal that would see the hostages released in exchange for the American-made military spare parts Iran needs to continue its fight against Iraq.

Both Carter and Muskie denied that such a deal has been made or is being negotiated. But, using language that continued the conciliatory signals the United States has been sending to Iran the last few days, both made clear that they are ready to discuss the hostage crisis with Rajai if he is willing. t

However, in the early days of the Iran-Iraq war, when the Iranian parliament was establishing a commission to deal with the hostage issue, it specifically forbade "all direct or indirect contact" with the U.S. government. It is unclear whether this prohibition extends to Rajai's current mission.

Carter, who was campaigning in the Northeast, told reporters at the Hartford, Conn., airport: "We have no arrangements to meet with Mr. Rajai. But if he were amenable, we would continue to follow our practice of meeting with any Iranian official who, had the possibility of speaking authoritatively for their government."

Asked if that meant he personally was willing to see Rajai, the president responded, "Yes, myself. . . ." He also suggested that the United States could be represented in such a meeting by Muskie, Deputy Secretary of State Warren M. Christopher or any other administration official acceptable to the Iranians.

Later, however, White House press secretary Jody Powell told reporters on board the president's plane that Carter saw "no real possibility" of a meeting with Rajai.

A similar tack was taken by Muskie while answering questions after a speech to the Woman's Natioinal Democratic Club here. Muskie recalled that he wrote to Rajai a few weeks ago urging a meeting or other contact regarding the hostages, and he also noted that the offer had been repeated two weeeks ago when he met with Iraq's foreign minister at the United Nations.

"The door is open to such discussions," Muskie said. "Whether or not such discussions will materialize I cannot predict."

Muskie also went out of his way to deny the spare-parts-for-hostages rumors that have been cropping up during the last week. At one point, he said, "There is no such proposal, no such deal. The reports are totally false." Later, he added, "I will answer the question precisely. No!"

But, while he sought to stress that there is no foundation for the rumors, Muskie did drop one strong hint that if Iran is willing to cooperate on the hostage issue, the United States might look with sympathy on its efforts to find spare parts for its American-made planes and other weaponry.

When a reporter asked what would happen if Iran released the hostages, Muskie replied: "If the hostages were returned, the sanctions [against Iran] would automatically go off. . . . The effect of that would enable Iran to reduce its isolation and perhaps have access to parts that it would not otherwise have access to.

"Now, should we therefore become part of Iraq's war aims and insist on the retention of sanctions in order not to be unneutral with respect to Iran? Think through that."

The comment appeared, on its face, to be the broadest signal yet that the United States is edging away from the position of neutrality it proclaimed when the Iran-Iraq war broke out last month, and is trying to telegraph to Tehran the possibility of U.S. backing if the hostage problem can be resolved.

Evidence of a U.S. tilt in that direction started becoming unmistakable Wednesday when Carter and his national security affairs adviser, Zbigniew Brzezinski, went out of their way to publicly volunteer the view that the United States backs the national integrity of Iran and considers that Iraq started the war by invading Iran as a means of resolving a border dispute.

That line was continued yesterday by Carter during his talk with reporters in Hartford. Although the president restated that the United States is neutral in the conflict, he added that the dispute "should not be settled by aggression or invasion of another country."

These persistent hints from the president and his top aides seemed to underscore that the administration sees the presence in New York of Rajai, the highest ranking Iranian official to visit this country since revolutionary forces took over Iran in January 1979, as a new opportunity to impress on Iranian leaders how isolated they are in the international community.

Rajai is associated with the hard-line Islamic fundamentalist bloc within the Iranian power structure, and it is this group that consistently has blocked attempts by the more moderate civil authority headed by President Abol Hassan Bani-Sadr to settle the hostage issue.

The apparent hope within the administration is that Rajai's experience at the United Nations will demonstrate to him the widespread hostility directed against Iran at a time when his country is in peril, and perhaps make him more amenable to negotiation.

However, it is not clear whether Rajai and other Iranian officials are prohibited from contacting U.S. representatives while in New York. Thus, there are continuing questions in U.S. policymaking circles about whether the opportunity afforded by his trip will prove to be illusory.

As Muskie said yesterday, "I have no way of knowing what his objectives may be beyond the stated one of participating in the Security Council debate on the war. Whether he has objectives that may impact on the hostage issue is something we have yet to determine."