Shiite leader Nabih Berri today set a new condition for the release of 40 American hostages held here: the withdrawal of U.S. warships from positions near the Lebanese coast.

In interviews with local reporters, Berri said this new demand was being made in defense of Lebanon and not in the name of the hijackers who originally seized the passengers on a TWA flight 11 days ago. He made it in the name of his Amal militia, he said, which has acted as intermediary in the current crisis.

Amal today released a videotape showing a dozen of the hostages. Although the film and the questioning of the hostages by their captors were apparently meant to show that the men were well and wanted to go home, the hostages clearly were controlled in their responses, appearing grim and suggesting restrained anger. Details, Page A11

The tape, made available to the four major U.S. television networks, was reportedly filmed Friday. However, there were unconfirmed reports that it was made a day earlier, the same day that five of the hostages appeared at a press conference at the Beirut airport.

The implied threat of U.S. warships reportedly passing close to Lebanon, Berri said, "makes us add another condition for returning the plane's passengers -- this time from the Amal movement -- namely that the American fleet sail away from our coast."

In Washington, Secretary of Defense Caspar W. Weinberger said the U.S. ships were in international waters, but Berri did not phrase his demand in specific terms and called for the flotilla to leave the area off the Lebanese coast generally.

Seven U.S. ships are reported to be between Lebanon's claimed 12-mile territorial limit and about 60 miles off the coast.

Until today, the central demand to be met before the American hostages would be freed was the release of all Lebanese held in Israel's Atlit Prison.

Berri rejected U.S. and Israeli contentions that Washington does not have the right to pressure Israel to release the detainees at Atlit. Berri, who is justice minister and state minister for southern Lebanon in the flimsy coalition government here, noted a 1982 pledge by U.S. envoy Philip Habib to the Lebanese prime minister committing the United States "to safeguard detainees held by Israel and implement international agreements related to them."

Although this pledge was made under different circumstances, Berri insisted that he still considers it in effect.

In Washington, a State Department official called Berri's statement about the 1982 pledge "a bit disingenuous." Habib conveyed guarantees undertaken by Israel, not the United States, and was referring specifically to Palestinians being evacuated at the time and their dependents, the official, who declined to be identified, said.

Israel, by moving Lebanese detained during its occupation of south Lebanon to a prison in Israeli territory, has violated the Geneva Conventions, Berri said. As such, the United States should call on Israel to release the detainees "whether because of a plane incident or without it."

Israel has denied violating the Geneva Conventions, citing other sections.

At the airport today, gunmen aboard the hijacked plane summoned a doctor to attend to one of the American crew men, identified by United Press International as Capt. John Testrake, who was said to be suffering from stomach pains. The airport physician, Dr. Albert Zbouni, boarded the plane escorted by Amal officials. He came out 10 minutes later but did not speak to reporters.

"Tell the doctor to keep his mouth shut. He should not talk to the press," a gunman in the cockpit told the control tower, according to The Associated Press. "Otherwise we'll cut his tongue out. I'll make his wife a widow."

Another top Shiite official said today that there has been "absolutely" no movement toward the release of the 40 American hostages here in the last 48 hours, despite the Israeli gesture of releasing 31 Lebanese prisoners.

Aked Haidar, head of the Shiite Amal movement's politburo, ruled out any reciprocal gesture by the hijackers who seized the Americans or by his own people.

Haidar told foreign reporters crammed into a locked bomb shelter for his press conference that after 11 years of war, the hijackers "have a lot of time; these people are very patient. We are very patient."

He added, however, that he hoped he was talking "about days, not about years."

"The solution is to release everybody," Haidar said. "How to schedule it, that is the problem."

Berri had dismissed Israel's release of the 31 prisoners as showing "no good will" on the part of Israel.

Haidar added that the negotiations are not "like buying a house on credit" with a down payment and monthly installments. The position of the hijackers and Amal remains all or nothing.

The 31 freed by Israel today arrived early in the afternoon at the southern city of Tyre in a yellow school bus to an enthusiastic reception by their families. But Israel continues to link the release of the rest to conditions in the security zone that it has set up under the control of a proxy Lebanese force, the South Lebanon Army.

Haidar rejected any possibility of reaching an agreement with Israel that is not guaranteed by Washington.

"We don't believe . . . a single word that comes from the Israelis," Haidar said. "We do not want dialogue with the Israelis."

If the United States makes an offer to set a precise date and time for the release of the prisoners in Atlit in exchange for the hijacked hostages, Haidar said, "then we will consider it." He suggested that tacit understandings would be unacceptable.

Before the conference, a rocket-propelled grenade lobbed from across the nearby Green Line dividing east and west Beirut landed about a block and a half away. Inside, security precautions were simple but effective. After everyone was searched, all were locked in with Haidar, and no one was allowed to leave the room until he did.

An aristocratic, gray-haired former colonel in the Lebanese Army, Haidar looked vaguely incongruous in his suit and tie sitting before posters of martyred boys and women and a sketch of Lenin. But he appeared unruffled by questions about the U.S. military forces in the region. He did not repeat Berri's new condition, which was unknown to the international press at the time.

"We know the muscle of the United States," Haidar said. "Everybody knows the Marines can take over Lebanon in one day's time."

"But President Reagan can't really release the hostages. He would pass over their dead bodies," Haidar said.

As Berri has done, he tried to put as human a face as possible on the crisis. The hostages, he repeated, are being treated well. They are having fun, Haidar insisted to an audience that did nothing to hide its skepticism. They were watching television, listening to radio, going to the beach and eating food from the best restaurant in Beirut, he added.

"We are not trying to convince them to become Shias," Haidar said. "We are trying to teach them only that there is another civilization in the world, and you have to respect it." Asked flatly if holding tourists hostage was representative of that civilization, Haidar hesitated a moment. "That is an interesting question," he said. "Certainly not."