In attacking Libya yesterday, the Reagan administration cited as justification the principle asserted by Secretary of State George P. Shultz in January that the United States has a legal right to use military force against states that support terrorism.

In a Jan. 15 speech at the National Defense University, Shultz said that under international law, "a nation attacked by terrorists is permitted to use force to prevent or preempt future attacks, to seize terrorists or to rescue its citizens, when no other means is available."

His argument was disputed then by many experts on international law. Several of those queried last night said it was not immediately clear whether the administration can apply that argument to Libya in a way that the international community will regard as legally sound.

At issue, the experts agreed, is whether the U.S. action fits the definition of self-defense in the United Nations Charter. The charter outlaws the use of force in relations between states, but Article 51 provides an exception when a nation is defending itself against attack.

In their statements last night, Shultz and President Reagan asserted that the United States had done precisely that. While accusing Libya of ordering the April 5 Berlin night-club bombing that killed an American soldier and injured others, they carefully denied that the U.S. action was "retaliation" -- a motive that administration legal experts acknowledge is not permitted by international law.

Instead, Reagan and Shultz argued that the United States had undeniable evidence that the Berlin bombing was part of a concerted Libyan strategy to harm the United States by attacking its citizens and property in several parts of the world.

Shultz said that the strikes against Libya were a "measured" and "proportionate" attempt to deter Libyan leader Muammar Qaddafi from continuing on that course.

Among those backing that position was Lloyd N. Cutler, who served as White House counsel under President Jimmy Carter and recently has written and spoken extensively about the legal aspects of intervention.

"I don't have any question about the legality of a measured military response as long as we had satisfied ourselves that the Libyan state was involved" in the bombing, he said.

Cutler acknowledged that "arguments can be constructed the other way" on the grounds that Article 51 normally is viewed as applying to self-defense against attacks on a nation's territory.

"However, as a superpower with global responsibilities, if our forces are attacked in another country, you can construe it as an attack on our territory, and we are entitled to make a measured response," he said.

A more cautious view was expressed by Oscar Schachter, professor of international law at Columbia University and former chief legal counsel of the United Nations. He said the right to respond militarily requires a more stringent test than whether U.S. citizens were attacked.

"If American tourists are killed by bandits in Mexico, that is not an attack on the United States, and it would be a very far-reaching proposition to extend the doctrine of self-defense to our attacking Mexico," he said. "Our government would have to be able to make the case that the foreign power is engaged in a concerted plan to attack the United States."

To satisfy the requirements of international law, Schachter said, the United States would have to establish that its actions were not punitive reprisal, were necessary to meet the threat to U.S. interests and were proportionate to the offense.

Reagan and Shultz pointed out last night that the United States had tried to deter Libya's alleged terrorism with economic sanctions, diplomatic activity and lesser shows of military force.

They said the United States escalated its response only after obtaining what Reagan called "direct, precise, irrefutable, solid evidence" that the Berlin bombing was part of a Libyan terrorism campaign that included planning for other attacks, some of which the United States forestalled.

Other experts said that the United States' ability to make a credible legal case will depend on how much evidence is made public and whether it clearly indicates Libya's culpability.

Monroe Leigh, State Department legal adviser under former secretary of state Henry A. Kissinger, said: "I think the president has to come up with his evidence, and the people will have to judge how good it is."

The experts said they did not have enough information to know whether the administration satisfied the 1973 War Powers Resolution, which permits the president to send combat troops into battle or areas of "imminent" hostilities only for 60 days without a declaration of war or a congressional mandate.