Four American hostages in Lebanon, saying their captors are "growing impatient," appealed to President Reagan today to abandon "quiet diplomacy" and negotiate with their kidnapers for their freedom.

In an open letter to Reagan, the hostages -- Associated Press bureau chief Terry A. Anderson, Thomas M. Sutherland, dean of the agriculture school of the American University of Beirut, David P. Jacobsen, administrator of the American University Hospital, and the Rev. Lawrence Martin Jenco, head of Catholic Relief Services in Lebanon -- said they had been told that a fifth hostage, U.S. Embassy political officer William Buckley, is dead.

There was no word about a sixth missing American, Peter Kilburn, librarian at the American University.

In Washington today, White House spokesman Larry Speakes reiterated the U.S. policy of not "negotiating" with terrorists, saying Reagan's stance "will not change." Details on Page A20.

Islamic Jihad, a shadowy Shiite fundamentalist group that has claimed responsibility for abducting the Americans as well as other westerners, has demanded the release of 17 Arabs held in Kuwait following the December 1983 bombings of American, French and Kuwaiti installations there as a condition for freeing the hostages. Five persons were killed in those attacks.

Kuwait has refused to release any of the 17, and it is not clear what effect any U.S. negotiating effort might have on Kuwait.

The message to Reagan was included in a packet dropped by a young man outside the building housing the AP offices in west Beirut. The letter to Reagan, including a statement by the four that conditions of their captivity were deteriorating, pointed to an increasing sense of urgency on the part of their Moslem extremist captors, who released another American, the Rev. Benjamin Weir, on Sept. 14 to press on the Reagan administration the necessity for negotiations.

The sealed blue folder delivered today also included letters addressed to the hostages' families, two U.S. congressmen, the archbishop of Canterbury, AP and other news media.

Pointing to the successful negotiations over the American hostages on the TWA plane hijacked in June and to efforts made by other countries, they said:

"You, and they, did so because you believed that saving the lives of innocent hostages should be the primary goal. We are asking for the same consideration. There is no alternative."

The letter also said "we are told" that Buckley is dead. Islamic Jihad announced on Oct. 4 that Buckley would be executed in retaliation for an Israeli air raid on the headquarters of the Palestine Liberation Organization outside Tunis and later released a photo of what it claimed were his remains. U.S. officials said the photograph did not conclusively prove Buckley's death.

The signatures of at least two of the senders of today's letters, Anderson and Sutherland, were positively identified by family or colleagues.

The communications, which carried today's date, came a day after callers claiming to represent Islamic Jihad said the six American hostages had been executed and their remains left underneath a factory in west Beirut. No bodies were found. One of today's letters acknowledged reports of the calls, saying, "obviously this is not true. Our captors say it was an attempt by the U.S. government to spoil negotiations."

In the letters made available to journalists, the hostages did not shed any light on their whereabouts. Of their confinement, they reported that "we are kept in a small, damp room 24 hours a day without proper exercise, sanitation, fresh air or balanced diet."

In an apparent effort to mobilize public support for efforts to secure their release, they asked their families to release all but the most personal parts of the letters to them and specifically asked AP not to heed requests from U.S. officials not to publish the letters. "We feel strongly that secrecy is delaying our release, and public discussion will speed it," they continued.

In Beirut, an elated Jean Sutherland, wife of the American University administrator, said after reading her husband's letter that he was "bearing up very well and he does mention that they the four captives are trying very hard . . . to keep themselves going. Hanging in."

"I think there is hope now," she added.

Jean Sutherland, who has been traveling between London, the United States and Beirut to make contacts and aid efforts for the hostages' release, said that Vice President Bush told relatives of the kidnaping victims in September that "they were ready to talk," an apparent reference to talks with the kidnapers.

Signaling that their captors were growing impatient with Washington's refusal to negotiate and that they will not give in to pressure from Syria, Iran or local Shiite leaders, the four said: "They will not be moved, and are growing impatient."

Addressing Reagan, they added: "You have tried other routes but have not won the release of a single hostage in more than 18 months. We have no chance of escaping and our captors say if any attempt is made to rescue us, they and we will all die."

Thanking the president for his efforts "through these long months" and noting that "your 'quiet diplomacy' is not working," they added, "Do you know the consequences your continued refusal will have for us? It is in your power to have us home for Christmas. Will you not have mercy on us and our families and do so?"

In a separate letter to Reps. George M. O'Brien (R-Ill.) and Robert K. Dornan (R-Calif.), they pressed their point, thanking the legislators for previous efforts on their behalf and urging them and other members of Congress to "try to persuade President Reagan to take the only course available to win our release and to take it quickly." The four requested that their letter to the archbishop of Canterbury, the Rev. Robert Runcie, be kept confidential and that a copy be transmitted to U.S. officials.

Syria has said that it would spare no effort in obtaining the release of the missing Americans along with other westerners abducted at gunpoint in Beirut during the past year. Despite the hostages' insistence that their kidnapers are not linked to Iran or Syria or subject to their influence, observers in Beirut said that tension between Tehran and Damascus could be impeding efforts to resolve the hostage crisis, particularly as concerns the American captives.

Intra-Arab efforts at rapprochement among the rival governments of Syria, Jordan and Iraq have strained ties between Iran and its Arab ally, Syria. Libya also backs Tehran in its Persian Gulf war against Iraq.

A Syrian drive to forge a Lebanese national reconciliation based on political reforms among the hostile communities has given rise to expectations that Syria may move to crack down on Moslem fundamentalist groups believed responsible for the abduction of westerners here.

Some observers here said that the timing of the letters delivered today indicated increased urgency on the part of the kidnapers to resolve the crisis on their own terms before any action against them by Syria or by its armed local allies.