President Reagan last night rejected generalized retaliation against the hijackers of Trans World Airlines Flight 847, saying it would be "an act of terrorism in itself," and declared his willingness to wait out the captors of the American passengers held hostage in Beirut.

"I have to wait it out as long as those people are there and threatened and alive, and we have a possibility of bringing them home -- I'm going to say a probability of bringing them home," a grim-faced Reagan said at a nationally televised news conference from the East Room of the White House.

After opening with a statement declaring that "the United States is tonight a nation being attacked by international terrorists" and reaffirming that "America will never make concessions to terrorists," Reagan spent much of the news conference explaining the difficulties of military retaliation in this kind of situation.

He said that "striking a blow in a general direction" would be an act of terrorism that could result in the "killing and victimizing of innocent people."

In ruling out concessions to terrorists, Reagan also declined to pressure Israel for refusing to release the more than 700 Shiite prisoners, whom the hijackers are demanding be turned loose before the Americans are freed. Reagan acknowledged that Israel may have violated international law in taking the prisoners out of Lebanon. However, he said, "the linkage that has been created makes it impossible for them and for us" to trade them for the hostages.

Instead of talking about retaliation against any group in Lebanon, where 40 American passengers and crew members are being held at undisclosed locations, Reagan directed his opening statement at the Greek government. He said he has directed the secretary of state to issue an immediate advisory for U.S. citizens traveling through Athens International Airport, where Flight 847 was hijacked Friday, warning of the dangers to them.

He also said he has appealed to U.S. air carriers "to review the wisdom of continuing any flights into Athens until the security situation there has improved."

Reagan also said in his opening statement that he was urging that no American enter any Middle Eastern country "that does not publicly condemn and disassociate itself from this atrocity," a warning that if taken literally could keep Americans out of many countries with which the United States is allied.

The president opened the news conference by paying tribute to Navy diver Robert D. Stethem, 23, of Waldorf, Md., who was brutally beaten and shot to death by hijackers early Saturday. Reagan called him "a young American hero . . . returned to his native soil in a coffin after being beaten and shot at point-blank range."

Earlier in the day the president called Stethem's mother, Patricia, and told her, "All Americans were shocked and deeply saddened by the death of your son. His courage and pride are an inspiring example to us all."

Repeatedly, Reagan was asked last night to compare his handling of the present situation with his sharp criticisms of President Jimmy Carter on the subject of the Iranian hostage crisis during the 1980 presidential campaign. On Oct. 21, 1980, Reagan said, "I believe this administration's foreign policy helped create the situation and the fact that they've been there so long is a humiliation and a disgrace."

Each time the question was raised, Reagan responded that the present situation is different because the hostages are not being held by a government that can be held accountable. When a reporter observed that Nabih Berri, leader of the Shiite Moslem movement known as Amal, who has said he has custody of most of the hostages, is Lebanon's minister of justice, Reagan replied that this identification "is to give the government of Lebanon a cohesiveness it doesn't have." "He has his own militia; he has his own army," Reagan said. "So, it isn't that simple that you can say this is the government of Lebanon."

Earlier in the day White House spokesman Larry Speakes appeared to criticize Berri after he had announced in Beirut the release of three of the hostages, one of them a Greek. Speakes said that "piecemeal exploitation of the captivity" of the hostages was "uncivilized behavior in its worst form."

But Reagan mostly refrained from criticizing Berri, whom the United States is counting on to play a pivotal role in winning freedom of the hostages. The president did say, snapping his fingers, that Berri "could be the solution to the problem that quickly."

Reagan's tone changed, however, when asked whether he would hold Berri responsible, if Berri returned the hostages to the hijackers if Israeli-held prisoners are not released. Berri has threatened to do this.

"Yes, I would," Reagan said.

While ruling out generalized military action, the president on several occasions expressed frustration over the situation facing the United States in trying to free the hostages.

Asked what he would say to those who think that America can no longer protect its citizens abroad, Reagan responded: "These people, I think . . . are jumping to conclusions and don't realize what the situation is. But I'm as frustrated as anyone. I've pounded a few walls myself when I'm alone about this. It is frustrating. But, as I say, you have to be able to pinpoint the enemy. You can't just start shooting without having someone in your gunsights."

Reagan's statement about the legality of Israel's action in taking the Shiite prisoners into Israel was a reiteration of comments made earlier in the day by his spokesmen at the White House and State Department. Speakes said the United States "firmly believes" that the prisoners will be released if the passengers are freed and said the hijackers were delaying this release.

But the president, asked why he didn't "lean on Israel a little bit," said the Israelis were unable to release the prisoners under the demand of a direct exchange by the hijackers.

"There was no question but that they were going to [release the prisoners] in stages . . . ," he said. "But it has now been tied to where such a movement would be, in effect, giving in to the terrorists. And then, as I say, who is safe?"

The president said his goal was "the safe return" of the hostages "in a manner that does not reward the hostages for the crime they have committed because that gang would be out next week for another try."

Reagan declined, on grounds that the United States was "in the midst of a dangerous and volatile situation," to answer several specific questions about the hostage crisis.

He declined to include seven Americans who have been held hostage in Lebanon for as long as year as part of what the reporter who asked the question called "the current crisis in Beirut."

Reagan said these Americans are included "in every conversation we have with our people there" and that "we haven't given up on them." But he also said it was difficult to know the whereabouts of these Americans, which some administration officials have said apparently are held by factions other than the Amal Shittes.

The president said America was still "standing tall," a phrase he used frequently during his 1984 reelection campaign and that he didn't think that U.S. policies had contributed to the rise of anti-Americanism in the Middle East. But he acknowledged that U.S. participation in the Middle East peace-keeping force had helped make Americans a target and said "there is an anti-Americanism that is rampant there on the part of those who don't want peace with Israel."

The news conference was dominated by the hijacking crisis, but Reagan responded to questions several other topics.

With being specific about time or place, he expressed optimism about the prospects of a summit meeting with Soviet leader Mikhail Gorbachev. He also said he couldn't be entirely preoccupied with the hostage crisis, "great as it is," because of the importance of other issues, including the budget and his tax-overhaul plan.

Reagan's strategists have adopted a policy of "business as usual" in an effort to demonstrate that the president has not been hamstrung by the crisis. Today, the president will make the latest in a series of speeches on his tax plan in Indianapolis.

In the concluding question of his news conference, a reporter referred to a number of setbacks Reagan has encountered this year, including the hostage crisis, reductions in MX missile deployment and the president's controversial visit last month to a West German military cemetery in Bitburg.

Reagan has sometimes been called "the Teflon president" because his critics say he isn't held accountable for mistakes. The reporter wanted to know if "the Teflon's that covered your presidency has slipped off."

"I never thought there was any Teflon on me anyplace," Reagan replied in apparent good humor. "But we seem to have reversed the course in regard to the contras," he said, referring to tentative congressional approval of compromise legislation to provide nonmilitary aid to Nicaraguan rebels.

He then launched into a spirited defense of his visit to the German cemetery, with an implied criticism of news media coverage of that visit.

"And with regard to Bitburg, in spite of the efforts of some of you, from the very first, I felt it was the morally right thing to do," Reagan said. "And I'm pleased that I did it."