The Iranian government said today that hostages being held by Moslem militants in the 12-day-old U.S. Embassy seizure could be released if the United States agreed to accept the findings of an investigation into the affairs of Shah Mohammad Reza Pahlavi -- including his possible return for trial.

The latest Iranian announcement represented yet another softening in the official conditions for release of the hostages.

The new offer, however, was tempered by the student captors' refusal to accept earlier Iranian government proposals aimed at obtaining the release of all hostages except American white males. The government apparently advanced the proposals to show Iran's good intentions in response to charges that the captives were being mistreated.

Acting Foreign Minister Abol Hassan Bani-Sadr balanced today's conciliatory offer by announcing that because of Washington's boycott of Iranian oil and its freeze of Iranian government assets in the United States, Iran was no longer accepting dollars in payment for its oil.

If Iran is able to make this decision stick, it would mark the first time that an oil-exporting country had dropped the dollar as the standard means of paymet. The Organization of Petroleum Exporting Countries long has threatened to drop the dollar in favor of an assortment of currencies as a means of pricing oil, but has never been able to put the idea into practice.

In an interview, Bani-Sadr underlined the psychological as well as the economic impact of the decision to demand payment in German marks and Swiss and French francs.

"This will weaken the dollar," he said. "If all the other oil countries follow us," he said of the new policy initiated two days ago, after the United States boycotted Iranian oil purchases, "The dollar is finished."

libya, with whom Iran has yet to establish diplomatic relations after months of estrangement, has signaled its support in a message from its leader, Muammar Quaddafi. Libya also has called for a special session of OPEC to discuss the American measures against Iran.

Bani-Sadr stressed that the only remaining condition for the hostages' release was American acceptance of an investigative commission, which he added will establish the shah's guilt and persuade Washington to extradite him.

Explaining his watered-down conditions for the hostages release, Bandi-Sadr said Iran would be satisfied if the U.S. would consider the right of our people to consider the shah a criminal who deserved to be judged." rThat, he said, would involve U.s. willingness to accept an investigative committee lines to establish the shah's "possible" guilt with the proviso that Washington agree to its possible consequences, including extradition to stand trial.

Earlier conditions over the past 11 days have varied from the orginal unequivocal demand for the shah's extradition and American acceptance of its own interference in Iranian internal affairs to U.s. acknowledgement of the shah's guilt and its responsibility for returning his family's wealth to Iran.

Bani-Sadr also modified earlier conditions that Iran alone choose members of the proposed investigative committee by saying, "They must be neutral and just."

Because of the recent U.S. retaliatory measures it was "unfortunately too late," he said, to act on Ayatollah Ruhollah Khomeini's offer last week for the "extradition or expulsion" of the shah.

That apparently ruled out release of the hostages if the shah left the United States for Mexico or Egypt where he has been offered asylum.

However, informed sources said, Bani-Sadr failed today to persuade students at the embassy to release all but the American white male hostages.

Earlier today he and another member of the ruling Revolutionary Council separately told reporters the conditions of the hostages' detention would improve and hinted that American women and blacks and non-American hostages could be released in the next few days.

According to the State Department, as many as seven American women and one black Marine are being held.

The Foreign Ministry said women and clacks -- possibly also meaning Asians -- would be freed because "Islam has got a lot of respect fore women and we consider blacks to be oppressed people."

The efforts failed because the students said that such gestures would be "inexplicable to the people" two days after the U.S. boycott and freeze. The students also refused to permit dialy visits by a foreign ambassador to check on the hostages, as the two Revolutionary Council members had promised.

Although "I was against taking hostages from the start," Bani-Sadr said, "now we are confronted with a fait accompli.

"To free them would be a sign of weakness. "Without the American government taking a step would be impossible. Our public opinion would not stand it."

Khomeini, who announced tonight he was canceling all public engagements until Dec. 5 because he was feeling tired and unwell, told the minister, "I can't move in that direction contrary to public opinion."

In a radio broadcast, Khomeini castigated the Americans and said, "They rob us and freeze our money just like thieves." He said nothing about the hostges. Khomeini was said to be angered by foreign diplomats' reports of ill treatment of the hostages. The students have told Bani-sadr their captives were enjoying "total comfort."

The minister asked the students why they tie the hostages' hands, informed sources reported, and was told it was because the students had no weapons.

He reportedly told them it would be better to carry arms rather than blindfold or bind the prisoners. But the sources said the students replied that the hostages were not being mistreated.

If the United States and Iran succeeded in settling their differences, Bani-Sadr indicated, the dollar would again be included among foreign currencies to be used in paying for Iranian oil.

"We don't want to cut all our ties to the U.S.," he said. "If we do business with you we will have to pay you in dollars."

Reacting to the announcement that Iran is demanding payment for oil in currencies other than the dollar, oil analysts in Washington said there are ample supplies of German marks and Swiss francs to accomodate an additional $100 million or so worth of transactions each day. That is approximately the daily value of Iranian oil exports.

[However, if all OPEC members were to damand payment other than in dollars, they said, there could be a run on the dollar as buyers switched to other currencies to pay the $1 billion or so Opec countries get each day for oil exports.]

Critizing what he said were efforts to impose hegemony, Bani-Sadr said, "I am against the existence of superpowers, but since they exist it is not a good policy to weaken one and automatically strengthen another."

He said he had received a Soviet note assuring him that the Kremlin had not gone along with U.S. opposition to Iran's request for a U.N. Security Council meeting on its crisis with Washington. The Soviet delegation yesterday voted with other members of the Security Council to back the U.S. position.

In explaining his opposition to holding hostages in the American Embassy, he said that he and the student's had had disagreements.

"If it had not been for the hostages, we would have had all world public opinion with us," he said in implicit criticism of the students.

The broadcast that raised hope of U.N. mediation quoted the acting Iranian foreign minister, Abol Hassan Bani-Sadr, as saying: "The latest reports, of which we are not 100 percent sure, indicate that Waldhem has pressed a favorable review on our legitimate request [for a U.N. Security Council meeting].

" we conveyed the reports to the imam [Ayatollah Ruhollah Khomeini] who said that if Waldheim regards the Iranian nation's rights to be logical, there will be no obstacle to his visit to Iran."

U.N. officials warned that Walddheim would not visit Iran "if he is going to be lectured by Khomeini about how villainous the U.S. is," but only if he is assured that the visit would be productive.

Meanwhile, the 15 members of the Security Council spent the day in another round of consultations on the Iranian crisis, designed to delay the council debaet requested by Iran without formally rejecting it.