Israel announced yesterday that it will release 31 of its more than 700 Lebanese Shiite and other Arab prisoners, but the hopes that the move might help free the 40 American hostages in Beirut faded as Lebanon's principal Shiite leader called the action inadequate.

Further dampening the prospects for a speedy end to the impasse over the fate of the 40 Americans, U.S. and Israeli officials denied that Israel's action was intended to placate the hostages' captors.

Both sides seemed to go out of their way to stress that there is no quick end in sight for the drama that began 11 days ago when Shiite gunmen hijacked Trans World Airlines Flight 847. The hijackers have demanded that Israel free all the Lebanese prisoners it moved from southern Lebanon to a prison near Haifa before they will release the American hostages.

Yesterday's developments played out primarily over American television, as the leading participants on all sides used Sunday talk shows to appeal to American public opinion in an effort to win understanding and support for the positions they have staked out.

The first public announcement that Israel will begin releasing the Lebanese prisoners came on network television, and Lebanese Shiite leader Nabih Berri was seen shortly thereafter dismissing the move.

Secretary of State George P. Shultz, taking the lead for the Reagan administration, reiterated in strong terms the U.S. position that it will not bow to terrorist demands and will not ask Israel to do so. He repeated U.S. demands that the hijackers release the Americans without preconditions and said:

"The problem is not Israel, the problem is not some evil about America, the problem is the people who hijacked that plane, who murdered an American and who are holding the Americans there hostage. That's the problem."

Israeli Prime Minister Shimon Peres and Defense Minister Yitzhak Rabin, seeking to counter impressions in this country that Israel is being too stubborn in refusing to release its prisoners immediately, used their separate appearances on American television to announce the Israeli Cabinet's decision to release 31 Lebanese.

But, while Peres said "we feel as deeply and as strongly about American hostages as we would feel about our own," he argued that for Israel "to surrender to terrorist blackmail" would invite further terrorism and thus do a disservice to the United States.

"The United States and Israel, together and not separately, are confronting the danger . . . ," he said. "It doesn't matter who surrenders, Israel or the United States, we shall break the united effort to bring an end to this terrible and unbelievable acts of cowardly people . . . . " On the other side, Berri, whom the United States has said it holds responsible for the safety of the hostages, agreed that he saw no connection between Israel's action and the status of the hostages.

President Reagan, returning to the White House from Camp David, echoed Peres' claim that the Israeli decision to release the 31 Arab prisoners was made in accordance with Israeli action on a legal appeal from the prisoners and was not related to the hijackers' demands.

"That is not a matter of linkage, that is a matter of Israeli law, it is a problem of their own . . . . It has nothing to do with our hostages," Reagan said.

The president repeated the pledge he made at a nationally televised news conference Wednesday that he will not resort to a military response as long as the hostages remain captive.

Shultz, who was interviewed on ABC's "This Week With David Brinkley," was more circumspect about possible U.S. retaliatory moves. "I'm not going to speculate about what the president may or may not do," he said.

But, in the face of persistent questioning about the administration's lack of action, Shultz said:

"People who are constantly expressing skepticism about President Reagan's determination, and kind of daring him all the time as you're doing here, should be careful. There is a long history of the president's opponents underestimating him . . . he's a very nice man, a very agreeable man, a very pleasant man, but he's also very tough and very determined, especially where he sees the lives and interests of America at stake."

Shultz said, "I don't want to get in the position of blaming the press for this," but added that reporting about the movement of U.S. military units had hindered U.S. efforts to free the hostages. He said media reports helped abort an attempt to work out a prisoner exchange through the International Red Cross at the outset of the crisis when the airliner had been taken to Algiers.

"I think it is the opinion of many who were dealing with the terrorists when they were parked in the Algiers airport that the tremendous amount of press reporting about U.S. military forces probably caused the airplane to break off in the discussions that were going on in Algeria and move back to Beirut, which was not a service to this whole thing," Shultz said.

But the conclusion that Shultz appeared to be conveying was that the administration thinks that it would be a major mistake to give in to the hijackers' demands and make concessions directly or by putting pressure on Israel.

"The safety of the hostages is of great importance to us," he said. "As the president said Saturday , America is a family, and when one American is murdered, we all feel it. So they are very important.

"At the same time . . . there is the fact that if we accede to a connection here, as is being sought, then we invite people to draw all other kinds of connections and use a tactic of grabbing Americans and trying to force us thereby to do something or other that they want to have us do. That is against our interests, and we need to be clear about that fact."

Peres, who appeared on NBC-TV's "Meet the Press," and Rabin, interviewed on CBS-TV's "Face the Nation," sought to identify Israel with that view.

"We support completely the position taken by the president of the United States and its administration not to surrender to blackmail," Peres said. "We have a real admiration for the position of President Reagan and Secretary Shultz . . . . We feel exactly like them. And if we should do something which is contradictory to their own policy, it will become counterproductive."

Peres and Rabin reiterated that Israel, which has acknowledged holding 766 Lebanese prisoners, always has planned to send them home once the situation in southern Lebanon is stabilized.

"If all of a sudden it will become quiet, that will be one situation. If it will continue to be dangerous and tumultuous, that's another situation," Peres said. "Let's not forget those people who are detained by us were people who were actively engaged in acts of terror against Israel. These are not innocent hijacked persons. They are not hostages. They are people who were caught red-handed, and we have arrested them in order to prevent murder and death and destruction."

That assertion is disputed by Lebanese Shiites who say that many of their people now imprisoned in Israel were rounded up and taken across the border merely because of suspicion that they might cause trouble during the withdrawal of Israeli forces from southern Lebanon.

Berri, who was interviewed by telephone on "Meet the Press," summed up the Shiite position by saying: "We are interested in 731, not 31."

He also made clear that the three-member TWA flight crew, which remains on the airliner, and a group of about six passengers who were removed before the other passengers, are being held not by his Amal militia but by more radical Shiite forces that apparently planned and executed the hijacking.

"When I accepted to be the negotiator, we agreed with the hijackers that we would release together," he said. "I cannot release them all because they are not under my control. If I released those held by Amal , it could endanger the others. It would break my agreement with those who captured the airplane."