President Reagan, who came to office vowing "swift and effective retribution" against acts of international terrorism, last night acknowledged the limits of American military power in freeing the 40 hostages held in Lebanon.

In his 1980 presidential campaign, Reagan said that "I wouldn't stand there and do nothing" about the hostages being held in Iran. He described then-President Carter's handling of the crisis as a "humiliation and a disgrace." Directly and indirectly in 1980, Reagan sought to exploit Carter's difficulties with Iran and to portray them as signs of weakness in Carter's leadership.

But last night an uncharacteristically somber and subdued Reagan appeared to have confronted frustrations much like those that bedeviled Carter. Reagan responded much as his predecessor had done, emphasizing the importance of protecting the hostages' lives and the problems of military retaliation.

When he welcomed the Iranian hostages home Jan. 27, 1981, Reagan declared:

"Let terrorists be aware that when the rules of international behavior are violated, our policy will be one of swift and effective retribution. We hear it said that we live in an era of limits to our powers. Well, let it also be understood, there are limits to our patience."

Last night Reagan confronted a hostage situation that in some ways is more perplexing than the one in Iran. He responded by trying to distinguish this situation from that earlier episode, and even by trying to avoid calling the 40 Americans held in Beirut hostages. Once he referred to them as "so-called hostages." And he said that he faces many limits in using American power to respond.

Said Reagan: "Retaliation, in some people's minds, might just entail striking a blow in a general direction, and the result would be a terrorist act in itself and the killing and victimizing of innocent people."

Those who want a stiffer American response, he said, "are jumping to conclusions and don't realize what the situation is," and "You can't just start shooting without having someone in your gunsights."

Reagan's response last night was surprisingly similar to Carter's in his first news conference after the hostages were captured in Iran. Asked then about a military retaliation, Carter said: "We certainly do not want to be guilty of the same violation of human decency and basic human pratices that have proven so embarrassing to many of the Iranian citizens themselves."

Carter said then that "my deepest commitment" was to win release of the hostages through peaceful means. Last night, Reagan said: "That is the goal, the safe return . . . . Yes, I could get mad enough now to think of a couple of things we could do to retaliate, but I would probably be sentencing a number of Americans to death if I did it."

Reagan appeared last night to have come a long way, not only from his criticisms of Carter, but also from his tough talk after earlier terrorist attacks in Lebanon during his presidency.

When the U.S. Embassy in Beirut was bombed in April 1983, Reagan said: "Those who directed this atrocity must be dealt justice. And they will be." After the October 1983 bombing of the Marine headquarters in Beirut, Reagan said: "Every effort will be made to find the criminals responsible for this act of terrorism, so this despicable act will not go unpunished."

Reagan did resort to the use of force in Lebanon with shelling by the battleship USS New Jersey, aimed at Syrian antiaircraft batteries, after Navy reconnaissance jets drew fire in December 1983.

But, for the most part, he has not followed through on his threats of retaliation against terrorists, reflecting a sharp controversy in his administration over such use of military force. The debate has raged ever since the United States decided at the last minute in late 1983 not to join a French retaliatory bombing raid after suicide bombers attacked U.S. and French positions in Beirut. One of the reasons then, as well as now, was uncertainty about who would be killed.

"You have to recognize," Reagan said last November, "that you don't want to just carelessly go out and maybe kill innocent people. Then you're as bad as the terrorists."