U.N. Secretary General Kurt Waldheim unexpectedly left Iran this morning after his two-day mission to negotiate the release of the American hostages here showed no visible signs of progress and suffered a damaging setback.

Sources at the Foreign Ministry here said Waldheim, who had been refused an audience with Ayatollah Ruhollah Khomeini, departed from Mahrabad Airport.

Yesterday, he experienced an emotionally turbulent second day of his mission, narrowly escaping an onrushing crowd of demonstrators at a grave site for victims of the Islamic revolution and viewing a macabre scene of some of the revolution's casualties.

Earlier yesterday, the U.N. envoy committed the world body to investigating human rights violations under the deposed shah, a gesture believed to have been made in hopes of softening the hard-line stand of Iranian officials and student militants holding the hostages.

They have demanded that the deposed monarch be returned to Iran and that the alleged crimes of the United States in Iran during the rule of the shah be exposed.

Waldheim also met for two hours yesterday with Iranian Foreign Minister Sadegh Ghotbzadeh.

Waldheim chose a visit to 300 crippled and maimed Iranians, said to have been disabled by the secret police of Shah Mohammad Reza Pahlavi, as the forum to announce an investigation of the former regime.

As the secretary general stood stiffly on a raised platform at the old officers' club of the shah's secret police, dozens of people paraded past, waving their prosthetic devices and crutches in the air and shaking them at the tight-lipped diplomat while they chanted anti-American slogans.

At one point, a 3-year-old-boy, whose arms were cut off at the shoulder by SAVAK agents allegedly as a reprisal against his leftist father, was passed into the arms of the envoy. The youngster, clad in camouflage overalls, burst out crying, and Waldheim, smiling wanly, handed him back to his mother.

When it came time for Waldheim to speak, his hands were trembling.

"I wish to extend to all those people whom I see here, whether they're grown-ups or children, my warmest sympathy and my deep feelings for them," he said. "My heart is with them.

"We will inqure thoroughly into the violation of human rights by the previous regime. You may be assured that what happened under the shah's regime will be [the subject of] an inquiry."

The exact mechanics for setting up such an inquiry and the likely forum for an investigation were left unclear yesterday, but a possible precedent could be the U.N. investigation into atrocities allegedly committed in Uganda by former president Idi Amin.

Iranian officials remained silent on Waldheim's pledge, even though some moderate forces, including Foreign Minister Ghotbzadeh, have expressed interest in such an investigation. Ghotbzadeh, who has spent five hours talking with Waldheim since Tuesday, avoided the issue after last night's Revolutionary Council meeting.

Asked about the progress of the negotiations with Waldheim, Ghotbzadeh said, "The basic problem is the return of the shah and his wealth."

There was little reaction within the United Nations to Waldheim's pledge for a human rights investigation. One U.S. diplomat, who did not voice any initial objection to the proposal, noted that the U.N. Human Rights Commission is designed to conduct such investigations.

[But he also pointed out that Iran in the past had not indicated acceptance of that agency's role and had pressed for a full-scale "international forum" on abuses of the former shah and the United States in Iran.]

Although Waldheim has held only seven hours of talks with Iranian officials, there were earlier signs yesterday that his mission had reached its limits.

Last night, Revolutionary Council spokesman Hassan Habibi finally ruled out a Waldheim visit to Khomeini, the Islamic spiritual leader and de-facto head of state.

"The imam (leader) won't meet with him," Habibi said flatly, confirming earlier indications that the 79-year-old Khomeini had little interest in a mediation mission headed by the representative of an international body that he considers to be under the control of the United States.

With no chance of meeting Khomeini or the militant Islamic students who have physical control over the hostages and who daily condemned the secretary general's visit, Waldheim has run out of meaningful negotiating partners.

From the outset, Waldheim set modest goals for his mission, and after a frustrating first day, he was believed to have lowered his expectations. The U.N. Security Council last week voted to consider economic sanctions against Iran if Waldheim fails to make any headway in securing the release of the hostages by Monday.

In another effort to demonstrate interest in Iran's painful past, Waldheim yesterday visited the Behest Zahra cemetery, where thousands of Iranians were buried after the revolt that toppled the shah.

Soon after his helicopter landed on a paved surface inside the south Tehran cemetery, more than 200 Iranians at a grave site 500 yards away ran across the flat stone slabs of mass burials and mobbed the black Mercedes Waldheim had entered.

Shaking their fists and waving snapshots of dead relatives, the demonstrators closed in on the vehicle, preventing the U.N. envoy from getting out to place a wreath on the large raised pyre of Ayatollah Mahmoud Taleghani, the revered Tehran holy man who spent 10 years in the shah's prisons and died shortly after last February's revolution.

Fifteen armed militiamen managed to keep the crowd from jostling Waldheim's car, permitting the driver to speed away to the waiting helicopter. Waldheim, clearly shaken and grim-faced, got into the helicopter unharmed and flew off to the foreign ministry where he met with Ghotbzadeh for two hours.

Although many of the demonstrators were legitimate mourners who seemed to spontaneously charge the U.N. chief's car, others appeared to have come to the cemetery in a planned effort to disrupt the ceremony.

Several dozen young men, dressed in white shrouds that indicate their willingness to die for religious causes, walked a long distance from Tehran to the cemetery at its southern border, carrying Khomeini placards and chanting anti-American slogans.