Iran's Revolutionary Council has reached a stalemate in its efforts to set up an international tribunal to expose U.S. "crimes" here because prospective members have insisted on prior release of American hostages, it was announced tonight.

The spokesman for the Revolutionary Council, Hassan Habib, said Iran's ruler, Ayatollah Ruhollah Khomeini, has rejected demands by potential members that the hostages held in the U.S. Embassy here be freed before the convening of a grand jury-style investigation proposed by the council.

The names of those contacted by Iran to serve on the tribunal have not been revealed, but Habib said several people were included and all laid down release of the hostages as a condition for their participation. Sean MacBride, former head of Amnesty International and winner of the Nobel and Lenin peace prizes is reported to be among those contacted. He spent two days in Tehran last week for talks with Revolutionary Council members.

"Since the imam has expressed the view that the hostages cannot be released, we cannot accept the conditions set by some people invited to be on the grand jury," Habib said.

Foregin Minister Sadegh Ghotbzadeh said yesterday that Iran might scrap the idea of a grand jury investigation and move instead to hold trials for the hostages themselves if the United States continues to seek economic sanctions against Iran from the U. N. Security Council.

The council's admission tonight, combined with Ghotbzadeh's warning yesterday, raised doubts whether such an investigation can be organized. The council, after much debate, had agreed that the Americans held hostage here for the last seven weeks should serve as witnesses in an expose of U.S. relations with Iran under the shah and then be released rather than face individual trials on espionage charges as demanded by their student captors.

In a nation of separate and sometimes conflicting power centers, the council consistently has proposed more moderate solutions to the sensitive hostage issue, often clashing with the militant Islamic students in the embassy who have threatened to try the Americans as spies unless the shah is returned to Iran.

Today's admission of stalemate in plans for the grand jury came only one day after the students promised to give up some of their independence in the hostage negotiations and work more closely with the council in seeking a solution.

In past weeks, the students have bowed only to the wishes of Khomeini, who remains the final arbiter and from his office in the holy city of Qom, issues edicts resolving disputes between the students and goverment leaders such as Ghotzadeh.

Khomeini, however, has been forced throughout the crisis to divide his energy between the hostage crisis and sporadic uprisings in rebellious provinces, where ethnic and religious minorities are demanding increased autonomy in the new Islamic republic and are taking advantage of disorder in government to press their demands.

More provincial unrest was reported tonight. The official Pars News Agency said supporters of the Moslem People's Party loyal to Iran's second-ranking ayatollah wounded a Revolutionary Guard and captured 10 other Khomeini followers in Tbriz, the capital of East Azerbaijan Province.

Last night's outbreak of violence raised the possibility of more fighting in the Tabriz area. Clashes early this month pitted Khomeini's Revolutionary Guards against gunmen loyal to Ayatollah Kazem Shariatmadari. The Azerbaijani leader opposes provisions in Iran's new constitution giving Khomeni supreme power to rule the country and has been backed by his local followers who also seek increased self-rule for their region.

The Iranian Revolutionary Council meanwhile ordered Ghotbzadeh to make a formal protest to Moscow over the Soviet role in Afghanistan's last coup, spokesman Habib said.

The Iranian move, announced after an unusual Friday meeting of the council, reflected concern in Tehran that the Soviet Union may think the new Iranian leadership is too preoccupied with domestic problems to worry about developments along its borders.

Iranian authorities were reported to be watching the situation closely. The head of the Iranian Navy, Adm. Ahmed Madani, said Iran and Pakistan have consulted on the Afghan coup and that Iran is taking extra military precautions along its mountainous border with Afghanistan.

In Tehran, Rep. George Hansen (R-Idaho) held a news conference to field questions on his latest efforts to obtain access to the hostages and, eventually, a pledge for their release.

Hansen, who said he is acting on his own and financing his own travels, became the first American allowed inside the occupied embassy to see some of the hostages during an earlier trip to Tehran last month.

He described this trip as an effort to follow up on the first one but was unable to report any progress in gaining admission to see the students or even obtain appointments with Iranian authorities.