In a new phase of its effort in the Iran crisis, the United States went to court yesterday to demand the release of its diplomatic personnel who have been held hostage in Tehran for 26 days.

The U.S. petition to the Internatinal Court of Justice at The Hague asked for an urgent hearing and a speedy legal judgment that Iran must release the hostages immediately and clear the American Embassy compound of unauthorized persons.

There was no certainty that Iran's revolutionary Islamic authorities or militant students would obey such an order by the eminent jurists of the world court. Proceedings could take weeks or months.

State Department officials indicated that Washington's main hope is to add legal pressure to the diplomatic pressures, moral suasion and the suggestions of future military action that are being brought to bear to protect the hostages from harm and obtain their release.

It is important for the United States to exhaust every peaceful remedy available, legal and otherwise, to convince Iran to respect international law, reporters were told.

The U.S. theory is that "at some point Iran must come face to face with the reality that it is isolated, that it is legally and politically at fault and morally at fault" and therefore release the hostages, said State Department spokesman Hodding Carter.

The legal action, which had been under consideration for more than a week in the high councils of the administration, was the newest element yesterday in the long-running crisis. Among the other developments:

The deposed shah, Mohammad Reza Pahlavi, has completed his current round of medical treatments in a New York City hospital, and was to have departed from there for Mexico at 3 a.m. today, according to informed sources. The unnusual hour apparently was chosen for security reasons.

Mexico's announcement last night, that the shah is no longer welcome to return there, added a new complication. However, Egypt's invitation to the shah is still open, according to Egyptian Ambassador Ashraf Ghorbal.

Several press reports earlier in the day mistakenly said a chartered Mexican turboprop had been standing by at New York's Kennedy International Airport to transport the shah. The plane turned out to be a seatless, slow-moving cargo aircraft picking up electrical equipment for the Mexico government's power authority.

A resolution was filed by 54 House members declaring that the embassy takeover is "a clear act of war" and proposing that the United States initiate "selected, deliberate, sustained and increasingly severe military operations against Iran" unless the hostages are released by a date to be announced in advance by President Carter.

Rep. Samuel S. Stratton (D-N.Y.), a sponsor of the resolution, said this admittedly "risky" plan is being recommended because "we don't feel time is necessarily on our side." Later one of the co-sponsors, Rep. Norman Dicks (D-Wash.), withdrew his name because of the president's expressed opposition to setting a deadline.

House leaders said the resolution will be blocked short of a vote. Chairman Clement J. Zablocki (D-Wis.) of the Foreign Affairs Committee called it "ill-advised and untimely" and said it "would jeopardize the lives of the hostages."

Rep. George Hansen (R-Idaho), who returned from a visit to Tehran during which he visited hostages and their captors, told a news conference it is unclear whether the militants at the embassy would free the Americans if their government told them to do so. Hansen repeated his call for a congressional investigation of the Iranian charges against the former shah, but seemed to back away from his earlier suggestions that such an inquiry could bring about the hostages' release.

Hansen said it is "absurd if not political" to charge that he was meddling in national policy by taking his free-lance trip to Iran. A public opinion poll of 405 people in his district that was sponsored by a local television station reported that more of his constituents opposed his trip than supported it.

In a rare joint statement, the national chairmen of both the Democratic and Republican parties said they back the president's effort in the Iran crisis, and said that only he can speak in negotiations for "a unified American people."

The U.S. petition asked the world court to act within days "in view of the extraordinary urgency of this case." A court spokesman said the 15 justices, from various parts of the world, are being called to The Hague immediately. It would be next week at the earliest that they could begin hearing the U.S. case.

The State Department spokesman said the United States is placed under no new restraint on its possible unilateral action while its petition is before the world court. If anything were to happen to the hostages during the period of the court hearing, sources said, the legal proceeding would not be likely to deter the Carter administration from taking the strong retaliatory action which has been hinted.

The U.S. appeal to the world court was filed in The Hague by State Department legal adviser Roberts B. Owen on the grounds that Iran had violated three international agreements on protection of diplomats, the 1955 U.S.-Iran Treaty of Amity and the U.N. Charter.

In addition to asking for legal orders against Iran, the equivalent of a preliminary injunction under U.S. law, Washington asked the president of the court, Britain's Sir Humphrey Waldock, to ask the government of Iran immediately not to place the hostages on trial, inflame public opinion against them or increase the danger to which they are exposed.

The basis for the lawsuit, which is highly unusual, if not unprecedented, is that Iran has violated fundamental principles of international law in not protecting the U.S. Embassy in Tehran, in supporting the actions of those holding the American hostages and in threatening to subject the hostages to trial despite their diplomatic immunity.

In addition to the hope for a quick and favorable judgment, according to spokesman Hodding Carter, the United States is filing the suit to drive home the point that Iran's action is unacceptable and "beyond the legal pale."

Carter said the United States continues its separate efforts to obtain a U.N. Security Council resolution calling for immediate release of the hostages, as well as iniitiatives in a variety of other channels.

The Security Council is scheduled to begin meeting Saturday night at the call of Secretary General Kurt Waldheim, who declared last weekend that the U.S.-Iran conflict is a serious threat to world peace. It was still unclear last night whether Iran will send a high-ranking representative, as it had promised earlier.

The world court is the principal judicial organ of the United Nations. Established by the U.N. Charter, the court can turn to the Security Council for enforcement of its orders if they are not voluntarily heeded by parties to suits before it. The Security Council has a variety of theoretical enforcement powers, but little practical power in the absence of unanimous agreement about what should be done.

The judges of the court, elected for nine-year terms by the General Assembly and Security Council, are currently from Britain, Nigeria, Poland, Senegal, France, United States, Soviet Union, Brazil, Argentina, India, Egypt, West Germany, Japan, Syria and Italy. Decisions are made by majority vote.

The U.S. petition confirmed officially that a private American businessman is being held at the embassy in addition to the 49 diplomatic personnel. The 50th hostage is Jerry Plotkin, employed by a California electronics firm. He was picked up by militants at a Tehran hotel.