As the latest Middle East hijacking drama assumed the character of a potentially protracted hostage crisis yesterday, the rumblings of the nation's collective frustration gave vent to a familiar cry: "Retaliate!"

But the calls for a retaliatory show of force, coming from some old faces in government as well as some newcomers, were muted somewhat when advocates were pressed on two perplexing yet poignant questions: Against whom should the United States retaliate? And would such an action deter, or only encourage and perhaps spread, the seemingly endless cycle of Middle East violence?

President Reagan, once an advocate of retaliation against terrorists, last night sounded a moderate tone when confronted with these same agonizing questions. "The problem is the who is perpetrating these deeds," he said at his news conference. "Who their accomplices are, where they are located."

Barely eight months ago, Secretary of State George P. Shultz warned that "there will not be time for a renewed national debate after every terrorist attack." Now the United States is once again playing the role of the confused Prince Hamlet, agonizing yet again over the thorny question of how to use force against terrorism.

The remarks Monday night of former secretary of state Henry A. Kissinger seemed to sum up the dilemma.

Appearing on ABC-TV's "Nightline" news show, Kissinger suggested the proper course for the Reagan administration was "no concessions, no negotiations, and retaliation when this is over."

Asked against whom the United States should retaliate, Kissinger backed away slightly from his earlier tough tone, saying: "I'm one individual, and we have a government with tens of thousands of people who can collect the intelligence as to who is most likely to blame. If they can't do that, we need a new intelligence system."

Kissinger insisted again that the United States take some retaliatory action, then added: "But what the precise action is, this involves a guessing game as to capabilities, intelligence information that I just don't possess. But that isn't my job. That's the government's job."

Reagan last night said the U.S. intelligence agents had "gathered a considerable body of evidence" about the terrorists, but he did not elaborate.

The first problem is identifying a target of the country's national rage. The list of suggestions yesterday ranged from Syria to Iran to the Shiite neighborhoods in the suburbs of Beirut, reflecting the deep uncertainty of the problem. Even Algeria was singled out, for not being tough enough in the crisis' early stages.

Several House members talked openly about a U.S. bombing attack on the Beirut airport, which seems firmly under the control of the Amal Shiite Moslem militia, whose leader, Nabih Berri, has adopted the hijackers' demands as his own.

"There's been some informal discussion on the House floor about possibly taking out the Beirut airport," said Rep. Bobbi Fiedler (R-Calif.). "That's certainly something I would suggest the president look at.

"It's important that we do retaliate in some fashion, when all of our citizens are back safe and sound," she said. "The American people are very angry at the moment and feeling very frustrated, as they were when there were hostages in Iran."

Rep. Robert K. Dornan (R-Calif.) offered another potential target: "Algeria must be made to pay a price" for not shooting out the tires of the plane while it sat on the tarmac in Algiers.

Perhaps summing up the frustration was Rep. Dante B. Fascell (D-Fla.), who asked: "Everybody wants to retaliate, but who do we retaliate against?"

His special assistant, Barbara D. Burris, added later: "We don't know that there's a government behind it. You can't just drop a bomb on a residential neighborhood and hope you hit the right people. That's the problem, that's the real conundrum of this whole thing."

On the Senate side, bipartisan emotions were equally strong in favor of using force, but there was just as much confusion as to the best target.

Sen. Thad Cochran (R-Miss.) said, "Somebody must be made to pay for this heinous, incredible act. We must demonstrate through action, not just words, that we will use our military resources to protect American lives and retaliate against those reponsible." When asked who that "somebody" was who must pay, Cochran's press secretary, John Perkins, conceded: "It's obviously a tough question."