A U.S. District Court judge ruled yesterday that Secretary of State Cyrus R. Vance did not have authority to revoke the passport of former CIA agent Philip Agee, an outspoken critic of U.S. intelligence policies.

Vance had taken the action against Agee in December, contending that Agee's activities abroad were "likely to cause serious damage to the national security and foreign policy of the United States."

The move followed reports that Agee, who now lives in West Germany, has suggested that the United States offer its intelligence files on Iran in exchange for the release of 50 American hostages in Tehran. There was also a published report, denied by Agee, that he had been asked to travel to Iran to participate in investigations of charges of illegal American activities there.

Agee's lawyer, Melvin L. Wulf, said yesterday that his client has "no intention of going to Iran or doing anything in connection with Iran until the hostages are released."

Justice Department attorneys are scheduled to appear before Judge Gerhard A. Gesell today and are expected to ask Gesell to postpone enforcement of his order restoring Agee's passport until the case is heard by the U.S. Court of Appeals. If Agee loses his passport, he faces deportation by West Germany.

Gesell ruled yesterday that the secretary of state's power to revoke or limit a passport is delegated by Congress which has expressed opposition and concern about restrictions on travel based on national security and foreign policy considerations.

Vance had relied on a 1968 State Department regulation, based on statutory authority to issue passports, which said passports could be refused when national security and foreign policy are threatened. Gesell, declaring the regulation invalid, said the statute does not give the secretary "'unbridled discretion' to grant, revoke or otherwise restrict passports."

Gesell noted that the regulation had been invoked only once in the 12 years it has existed.