Defense Secretary Caspar W. Weinberger, asserting that the United States reserves the right to retaliate militarily against terrorists holding 40 Americans hostage in Beirut, yesterday rejected Lebanese Shiite leader Nabih Berri's demand that U.S. warships be withdrawn from striking distance of Lebanon.

In an interview with network television reporters, Weinberger said U.S. naval and Marine forces, sent to the area after the hijacking of a Trans World Airlines passenger plane, are in "international waters" and not violating Lebanese territorial waters. Other U.S. officials said the ships are beyond the 12-mile limit claimed by Lebanon.

Weinberger was responding to the demand by Berri, leader of the Amal militia, that "the American fleet move away from our seafront" as an apparent new condition for release of the hostages. Berri has been trying to negotiate on behalf of hijackers demanding release of more than 700 Lebanese Shiites and other Arabs held in Israel.

Israel freed 31 of the prisoners yesterday. But Berri rejected the move as inadequate, and the Reagan administration has made a strong effort not to raise hopes of a speedy end to the hostage impasse. Weinberger echoed that line yesterday, saying:

"You can't really identify all of the turns and twists that this very unhappy episode might take. It is important to have the American presence there for whatever might be required of it. It is important, I think, to be close to wherever we may be needed whether we are needed or not."

Although President Reagan has said the United States will not retaliate while the hostages are captive, Weinberger said, "It's important for everyone to know that the United States has reserved its rights to take whatever action seems to be proper under the appropriate circumstances and those circumstances change quite frequently."

He said, however, that a military solution has been made "very much more difficult" because the hostages have been separated into groups and that the United States wanted to avoid injuring innocent Lebanese civilians.

White House and State Department officials rejected Berri's new demand, saying tersely that U.S. ships are "not in Lebanese waters."

That was in line with what administration officials privately described as their desire not to fan public frustration over what is expected to be perhaps a month or more of trying to wait out the hijackers and reach a solution that does not appear to be U.S. capitulation to terrorism.

The officials reiterated that efforts to find a basis for talks with Berri are "static" or "frozen." They particularly denied new rumors from Lebanon that Syrian President Hafez Assad, who has great influence over Lebanon's feuding religious and political factions, has acceded to Reagan's requests to intervene.

One official said Syria, which had encouraged terrorists as a means of forcing the United States out of Lebanon, "generally has been helpful" by "counseling the Lebanese toward moderation."

Diplomatic sources said a letter to Reagan from Israeli Prime Minister Shimon Peres was delivered yesterday. According to the sources, Peres repeated assertions he made Sunday on U.S. television of Israel's support for the U.S. policy of not acceding to terrorist demands.

The Israelis have cited the U.S. position in refusing to swap prisoners directly for the American hostages, but Peres has reiterated Israel's promise that it will release its Lebanese captives according to a timetable determined by Israeli security concerns.

A senior State Department official said Greek officials have made substantial progress in improving security at the Athens airport, where the hijackers boarded the TWA flight June 14. A subsequent U.S. travel advisory warning of a high risk of terrorism at the airport has confronted the Greek tourism industry with possible substantial financial losses. The senior official said the administration assured Greece that the advisory was not intended to be punitive. He added that the United States is not ready to rescind the advisory, but is encouraged by steps taken by Greek authorities for better monitoring of passengers and baggage and will reconsider when it believes that security has been sufficiently improved.

Vice President Bush, on a seven-nation European tour that will include discussing terrorism with U.S. allies, said in Rome, "We are not now, nor have been, or will ever be in the posture of knuckling under to demands."

The Rev. Jesse L. Jackson, who last year negotiated Syrian release of a U.S. Navy flier shot down over Lebanon, met yesterday with State Department officials and said he is willing to go to Lebanon or Syria "if it would help."