President Carter and his national security affairs advisers, Zbigniew Brzezinski, sent conciliatory signals to Iran yesterday on the eve of an unexpected -- and potentially important -- visit to New York by that country's prime minister.

Carter, in two town meeting appearances, and Brzezinski, on public television's MacNeil-Lehrer program, went out of their way repeatedly to volunteer the view that the United States backs the national integrity of Iran, which is under attack from neighboring Iraq.

Carter, while not departing explicitly from the U.S. stand that it is neutral in the Iran-Iraq war, at one point called Iraq "an invading nation," and in discussing the war at another point said that the United States opposes settlement of disputes "by aggression."

High U.S. officials are known to have been discussing for some time ways to send signals to Iran of U.S. sympathy and potential backing if the problem of the 52 American hostages can be resolved. The unsolicited remarks by Carter and Brzezinski appeared to flow from these considerations.

The remarks took on added significance in view of the scheduled arrival in New York today of Iranian Prime Minister Mohammed Ali Rajai. While declining formal comment on the Rajai mission, Washington policymakers described it as bringing new opportunities on both the war and the hostage questions. They added, however, that there is no indication that the various and often contentious forces in Tehran have made the hard decisions required to bring about an immediate resolution of either issue.

Rajai, whose arrival to participate in the United Nations Security Council debate on the war was announced late yesterday, will be the highest ranking Iranian official to come to this country since the revolution that overthrew the late shah, Mohammad Reza Pahlavi, in January 1979. U.S. sources noted, however, that there are continuing questions about the extent of the authority wielded by the 46-year-old former teacher, who is considered a hardline Islamic fundamentalist.

Since taking office last August, Rajai has made a number of hostile statements about the United States, and displayed no outward sign of eagerness to solve the problem presented by the hostages. In a speech in early September he ridiculed a conciliatory letter sent to him by Secretary of State Edmund S. Muskie.

However, U.S. officials expressed cautious hope that the sentiments of the world community on the matter, as expressed at the United Nations, will make an impression on Rajai, especially because he represents the hardline factions that have blocked past efforts to resolve the hostage problem through negotiations.

Iranian leaders who have been out of their country in recent months have come home more impressed with the country's isolation and the need to end it by freeing the hostages. The hope in Washington is that Rajai will receive the same message and be inclined to act on it in a period of Iranian national peril.

It is unclear whether Rajai will be authorized to discuss the hostage problem with U.S. officials while he is in New York. There is no indication, according to U.S. sources, that his mission here is to bring an end to the captivity of the 52 Americans.

Iran's willingness to participate at a high level in the Security Council debate on the Persian Gulf war was seen here last night as a potential beginning of a process of mediation that eventually could end the fighting. Officials said it is significant that both Iran and Iraq are willing to discuss the conflict in the U.N. forum, even though the positions of the two parties are far apart.

The expectation in Washington is that arranging and carrying out a negotiating process on the war between the two historic enemies will be difficult and time-consuming. But, in the U.S. view, the Rajai mission may provide at least the chance for a start.

Brzezinski, in his televised interview, strongly suggested that the Soviet Union is fueling a continuation of the war by providing or facilitating military supplies to both Iraq and Ira. He said "some military aid has been going to Iraq from Eastern European countries closely tied to the Soviet Union" while "Libya and North Korea appear to be providing arms or material to Iran, perhaps even transiting Soviet air space in one case."

State Department spokesman John Trattner, speaking earlier in the day, said the United States has no confirmation that the Soviets have provided "any large quantity of military assistance to either side."

Official sources said there are indications that "a trickle" of Soviet weaponry is going to Iran via a rail line between the two countries, apparently the latest installment of military supply contracts between Moscow and Tehran that date back to the reign of the late shah.

The Soviet Union has been the main source of Iraqi military equipment for a decade. But U.S. officials said that the supply to Iraq has been on a limited scale since the outbreak of the war with Iran.