The Supreme Court's Monday ruling in the case of former CIA agent Philip Agee was attacked by many constitutional scholars and lawyers yesterday as a potential licence for government restriction of speech and travel by political dissidents, journalists or anybody else.

The court upheld the Carter administration's revocation of Agee's passport for his open effort to destroy the CIA by exposing the names of undercover operatives.

Had the court limited itself to Agee or comparable situations, critics said yesterday, they would have had few objections. But they said the court went far beyond what was necessary to back up that action.

In the process, said Harvard Law Professor Laurence H. Tribe, the court "left a loaded gun aimed" at free speech and travel. The ruling "was a disastrous departure from doctrines protecting an open society," Tribe said.

"The sweep of the decision may be such as to encompass far more than errant CIA agents," said Floyd Abrams, a prominent lawyer in free-speech case. The scope, he said, is "breathtaking."

"It seems to me absolutely clear that under this opinion, if the Johnson or Nixon administrations had wanted to pull passports from reporters in Vietnam, this opinion would authorize it," said American Civil Liberties Union attorney Mark Lynch, who defeated former CIA agent Frank Snepp last year.

"Or if some disarmament person makes a speaking tour through Europe, saying it's a bad thing to deploy nuclear weapons there, the secretary could say it's inciting opposition to NATO" policy and revoke the passport, Lynch said.

Robert Dalton, State Department assistant legal adviser, said the government had no intention of using the ruling that way. And he said the context of the opinion, Agee's specific transgressions, might limit its application implicity. Jack Landau, of the Reporters Committee for Freedom of the Press, agreed with Dalton.

But Dalton acknowledged that nothing in the opinion explicitly limited the government to the facts in the Agee matter.

Dean Rusk, secretary of state during the Vietnam war and now a University of Georgia professor, said he doubts that the opinion gave such "sweeping permission" to the government. "The court will narrow it down to what appear to be real national security questions," he said.

If another war situation arose, Rusk said he would recommend a different way to handle damaging media coverage. "The Congress must consider the question of censorship. That was the first armed conflict ever fought on television. This is a problem in the realm of unfinished business," he said.

Chief Justice Warren E. Burger wrote for the court Monday that the secretary of state may deny a passport to anyone he determines may do "serious damage" to national security or American foreign policy.

He did not define "serious damage," but left the definition to the secretary of state. He did not limit the application of the ruling to CIA agents, former CIA agents or anyone else.

And in one of the two or three passages that most alarmed the critics of the opinion, the chief justice suggested that the government need not worry about claims that it is overstepping the Constitution. When there is a "likelihood of serious damage to national security or foreign policy," Burger said, "these claims are without merit."

What is "serious damage?" Dalton was asked. "It's a term of art," he said, undefined in the law or the regulations.

"The talisman of national security seems to have been invoked as a conversation stopper," Tribe said of the opinion. He said the ruling went well beyond any of the prior circumstances under which the court has allowed First Amendment restrictions, such as when there is an "imminent" or "clear and present" danger.

Instead, he said, "The mere fact that the intent of the speaker is to jeopardize American policy, which may be a fancy way of saying to 'change' American policy, is substituted" in the opinion.

Abrams said the decision was a greater threat to freedom of travel than freedom of speech. "It could be read to give the government sweeping authority to deprive citizens of a chance to travel abroad if their purpose or likely effect is 'damaging to foreign policy.'" he said.

Passports are required by law for travel abroad by U.S. citizens.