Legal experts said yesterday that the Ayatollah Ruhollah Khomeini and the Iranian militants have no basis under international law to put on trial as spies any of the American hostages held in the U.S. Embassy in Tehran.

"Every embassy has people who do commercial intelligence work and every embassy tries to talk to people and find out what is going on," said Seymour Rubin, executive director of the American Society of International Law.

"I would say there is absolutely no legitimacy for what the ayatollah and his people are threatening to do over there," he said.

Iranian militants at the embassy released three hostages late last night, Washington time, and there were indications that as many as 10 other captives would be released later. But in interviews with U.S. television reporters, Khomeini said some and possibly all the other hostages could be put on trial as spies if deposed Shah Mohammad Reza Pahlavi were not returned as the Iranians have demanded.

"The diplomats in any country are supposed to do diplomatic work, not the crimes of espionage," Khomeini said in an interview with ABC-TV. Khomeini did not elaborate on what activities he was referring to, but a Tehran radio broadcast yesterday said militants had found evidence of forged Iranian exit visas in the embassy.

Khomeini told CBS' "60 Minutes" that Iranian militants learned of espionage activities at the embassy after they took it over Nov. 4. The Iranian leader said in an interview with NBC's "prime Time Sunday" broadcast yesterday that his government would have to decide which embassy employes were spies.

"Those who have been normal workers in the embassy without espionage, they will be returned and the rest will be tried," Khomeini said.

According to experts, international law does not provide for trying diplomats suspected as spies in peacetime. They said the Iranian threat is without precedent.

"The proper response for the Iranian government would be to withdraw the accreditation of those persons engaged in that activity and give them X hours to leave the country," said Robert Goldman, dean of the American University Law School here.

Goldman, a professor of international law, noted that the United States expelled German and Japanese diplomats suspected of spying at the start of World War II rather than bring them to trial.

Iran, he said, has been seeking to get the United Nations to condemn the United States for haboring the shah, who is here for cancer treatment. "However," he said, "Khomeini has violated the minimum norms of the system that lead to the creation of the U.N."

Questions have come up over the years over whether diplomatic immunity extends to low-level foreign embassy aides, usually over situations such as traffic accidents in the host country, experts said.

However, they said that in the Iranian situation there appears to be little question about the diplomatic status of the hostages.

"My sense of what is going on in Iran is that the people who are most entitled to immunity are the ones most likely to be prosecuted," said Richard Falk, professor of international law at Princeton University's Woodrow Wilson School of International and Public Affairs.

Falk was invited to Iran by the revolutionary government last January after the shah was deposed. He said that the U.S. Embassy there was cited to him then as a symbol of espionage and repression for some Iranians.

"The wrongful use of the embassy extended beyond intelligence-gathering to various cover operations and to a relationship in training the Iranian secret police," he said.

"The latest actins in Iran," said Falk, "suggest that the stability of immunity may be closely connected with limiting the role of an embassy to legitimate functions."

Nevertheless, he said, the seizure of the U.S. Embassy in Tehran violated both the immunity of the members of the U.S. diplomatic mission and that of the embassy itself.

Although international law experts were unanimous yesterday in condemning the seizure of the hostages and Khomeini's threats of spy trials, they said the United States could undermine its protests with its efforts to deport Iranian students. President Carter last week ordered the Justice Department to deport those Iranians who have violated the U.S. laws.

"International law requires no discrimination against a certain class of aliens just because they are aliens," said Goldman. "Singling out the Iranians as a whole, depending on how the government undertakes prosecutions and deportation proceedings, could end up damaging our legal and moral position in this situation."