The Carter administration, facing potentially serious new threats to the 52 American hostages in Iran, said yesterday it would welcome a United Nations inquiry into the treatment of Iranian demonstrators being held in New York prisons.

The U.S. offer came after a barrage of charges by Ayatollah Ruhollah Khomeini and other Iranian authorities that the demonstrators, arrested during a violent protest here July 27, have been tortured.

In response, State Department spokesman John Trattner said: "I have found no evidence of torture or mistreatment. We would welcome an inquiry by the United Nations, just as we would welcome a similar inquiry into the treatment of our hostages in Tehran."

Underlying the administration's willingness to cooperate with a U.N. investigation, U.S. officials said privately, is growing concern here that the escalating confrontation over the demonstrators could cause major new complications and delays in the effort to free the 52 Americans held in Iran since Nov. 4.

The officials cautioned that there is no evidence at this time that Khomeini's revolutionary regime plans to use the arrest of the demonstrators as a pretext for some new move against the hostages, such as subjecting them to spy trials.

But, the officials added, the crescendo of harsh anti-American rhetoric coming out of Tehran in the wake of the arrests here could dash the administration's cautious hopes that Iran's chaotic political situation might be reaching a point of stability sufficient to allow a new try at ending the hostage crisis.

The administration has been clinging to the hope that a confluence of factors -- the death of deposed shah Mohammad Reza Phlavi, the convening of the Iranian parliament and the revolutionary regime's need to address pressing domestic problems -- might be nudging Iran's feuding political factions toward a consensus that it is in their common interest to rid their country of the hostage problem.

Now, U.S. sources said yesterday, there is concern that this momentum might be stopped by a new wave of anti-American hysteria introduced by the arrest and threatened deportation of the 192 Iranians who clashed with Washington police.

For that reason, the sources continued, the United States is sparing no effort to communicate to Iranian authorities -- through indirect diplomatic contacts and, if necessary, through a U.N. inquiry -- that the prisoners are not being mistreated and could be released with the right to remain in this country if they would only cooperate with U.S. law.

What is not clear, the sources said, is whether Iranian authorities will accept these reassurances or whether, through prejudice or deliberate design, they will treat the arrests as an excuse for prolonging the hostage crisis and attacking the United States in the international community.

The sources also said the administration does not know whether the demonstrators acted spontaneously or as part of some plot orchestrated from Tehran. But, the sources conceded, whatever the motivation, the subsequent refusal of those arrested to give their names or cooperate in other ways with the law has frustrated the official U.S. desire to treat them as leniently as possible.

The administration's "first wish," as one source put it, was that the prisoners would identify themselves, accept release on bond and remain in the United States. But, the sources added, given the prisoners' intransigence, there is general agreement among the agencies involved -- the State Department, the Justice Department and the Immigration and Naturalization Service -- that the only recourse is to handle them in strict accordance with the remedies prescribed by U.S. law.

According to the sources, the administration reluctantly felt compelled to institute deportation proceedings, partly in hopes that this might induce the prisoners to start cooperating and partly because it appears to be the most humane remaining method of dealing with them.

If the prisoners are deported, the sources pointed out, the United States would be able to argue that it had given them their freedom and sent them home -- in marked contrast to the way in which Iran has treated its American captives. Similarly, the United States also would be able to say that it had resorted to deportation only after giving the prisoners every opportunity to satisfy the legal requirements of remaining here.

However, the sources admitted, it is very unlikely that the solution will be that simple, because of the prisoners' ability to prolong the situation through legal appeals and because Iran or another country is unlikely to accept them.

For that reason, the sources said, the United States probably faces the threat of a drawn-out tug-of-war over the demonstrators that will give anti-American forces in Iran a continuing infusion of fresh propaganda ammunition to use against this country.