Ayatollah Ruhollah Khomeini today dismissed United Nations Security Council efforts to help settle the American-Iranian crisis and said that any investigation of Shah Mohammed Reza Pahlavi's alleged misdeeds must take place here and not abroad.

Khomeini thus doomed a number of proposed formulas designed to investigate the ousted shah's purported crimes either in the U.S. Congress or in international tribunals.

His latest hard-line message coincided with a fresh and detailed warning from radical Islamic students that they have mined and placed explosives in the walls, grounds and buildings at the U.S. Embassy where they are holding 49 American hostages.

Among those caught off balance by Khomeini's latest message was Acting Foreign Minister Abol Hassan Bani-Sadr, who had praised Rep. Henry Reuss (D-Wis.) for holding open the possibility of a House Banking Committee investigation if the hostages are released.

Reuss' initiative, Bani-Sadr said, was proof that his "round the-clock efforts to explain the facts to American and international public opinion have now borne their first fruit."

Reuss, responding to a cable from Rep. George Hansen (R-Idaho) requesting hearings on the shah's alleged crimes, said he might commit his committee to hold the hearings if they would help free the hostages but he would not proceed with such an inquiry until their release. Hansen visited Iran on his own initiative.

[Hansen accused the State Department of "pushing the self-destruct" button" by allowing the shah into the United States despite warnings of violent reprisals from Iran, the Associated Press reported from Tehran. Hansen, riled by Carter administration criticism of his unsanctioned intervention in the U.S.-Iran crisis, said Secretary of State Cyrus R. Vance must have been aware of the dangers. The congressman is scheduled to leave Wednesday after a week-long visit.]

Bani-Sadr urged Congress to "form other investigative committees to examine the crimes and treason of the former shah."

He added that the "scandal of the former shah and his American assistants will, by degrees, be greater than the Watergate scandal."

Khomeini criticized the Security Council for allegedly wanting to deal only with the hostage aspect of the crisis and not Iranian demands for the shah's extradition to stand trial here.

Convinced that the Security Council was "under direct U.S. influence," the 79-year-old religious leader said that its findings were prejudged and its membership "would welcome our oppressed nation's being found guilty."

"It is not possible to study the cases of the ousted shah and the hostages at the espionage center" -- as he now calls the U.S. Embassy -- "except in Iran because the evidence for the crime exists in Iran and cannot be transferred abroad," Khomeini said in a message from the holy city of Qom.

"We have given about 100,000 martyrs and have several million witnesses and more than 100,000 maimed people," he added, "whom it is not possible to take abroad to produce evidence to testify -- apart from the many files which exist here."

Khomeini argued that the "investigation of the espionage center must be undertaken at the same so-called embassy because it is there that the evidence of the crimes exists."

Iran's leader has threatened to put the remaining hostages on trial for espionage unless the United States extradites the shah.

The students occupying the embassy said they booby-trapped and mined the compound to thwart any attempt to take away the hostages, who are in their fourth week of captivity.

"U.S. agents intend to enter the spying embassy these days by any means possible and hijack the hostages or harm them," the students' message said.

Analysts suggested that the warning may have been motivated by fears that the extreme left was planning to stage disorders at the embassy during the Tasua and Ashura holidays, Shiite Islam's holiest. The holidays this year fall on Thursday and Friday.

"If the U.S. mercenaries try to carry out the plot," the message said, "they can be identified and will receive their punishment." The message made clear that Revolutionary Guards on duty at the embassy would shoot anyone suspected of such an attempt.

Giant marches are scheduled for both Tasua and Ashura. They were scheduled partly in commemoration of last year's massive demonstrations, which spelled the beginning of the end for the shah, and partly to show support for Khomeini in the crisis with the United States and to work up enthusiasm for the forthcoming constitutional referendum.

During the day a small group representing the pro-Moscow Tudeh Communist Party tried to march on the embassy, but were driven off by the Guard.

Spokesmen for Tudeh, which has come under increasing Islamic criticism although it strongly backs Khomeini, today defended their performance.

The party said it had "never disputed the important social role of the religious and has never denied that the religions of all systems [slavery, feudalism and capitalism] have on numerous occasions, because of their revolutionary and progressive contexts, been the rallying cry of oppressed nations."

In other developments:

The air space around Qom, Khomeini's residence 80 miles south of here, was "closed to all air movements by passenger or cargo planes," apparently as a precaution against any eventual American military intervention.

A group calling itself "the militant clergy" again charged the United States was responsible for the attack on Mecca's Great Mosque, Islam's holiest shrine.

Army headquarters announced a halt of military activity in Kurdistan to allow autonomy talks to move ahead between Kurdish leaders and a central government goodwill mission. Two days ago the Kurdish Democratic Party announced a 20-day ceasefire on condition that all non-Kurdish Revolutionary Guards were withdrawn from the western province within 15 days. r

Ayatollah Kazem Shariatmadari's office in Qom issued a statement contesting an interview the moderate leader gave to a Madrid newspaper. It said the newspaper's version was guilty of "great mistakes made in the translation" and denied "the remarks as they have been said."