The Reagan administration's efforts to secure the release of 39 American hostages held in Beirut through transfer to the custody of a foreign government appeared to run into some obstacles yesterday, but American and Israeli officials and the chief Moslem negotiator in Beirut expressed optimism that a deal could be worked out soon, perhaps by this weekend.

In Washington, Beirut and Damascus, there were also suggestions that the Syrian capital had emerged in the past 24 hours as the most likely spot for any negotiated transfer of control of the Americans from the combined group of hijackers and Amal Moslem militiamen who still hold them.

In Jerusalem, Israeli Prime Minister Shimon Peres told an American television interviewer yesterday that he believes "there is a fair chance for the safe return of the hostages. Israel," he said, "is trying to be as helpful as we can."

Israeli television reported late Thursday that the United States and Israel have reached an understanding that Israel will free its 735 Lebanese prisoners only after the 39 Americans held in Beirut are released, according to the Associated Press.

The television report carried no attribution. If the report is accurate, it would provide the first indication that Israel is willing to link its prisoner release with the release of the hijacked TWA passengers. Israel has said previously that it would free its prisoners gradually, in keeping with its security needs in southern Lebanon, and would have no linkage whatsoever with the hijacking of the U.S. jetliner. A U.S. official said that no actual agreement with Israel had been reached along these lines, but that indications had been received in Washington that Israel is shifting its position.

Aside from this report, however, there was no indication that the fundamental deadlock had been broken between the United States and Israel over finding a formula that would gain freedom for the American hijacking victims without compromising both nations' firm stand against negotiating with terrorists or giving in to their demands.

The key terrorist demand is that Israel free the Lebanese citizens, mostly Shiites, rounded up as possible security risks during Israel's withdrawal from southern Lebanon. They have not been charged with crimes but are being detained in Israel. The Reagan administration has said the Lebanese are being detained in violation of the Geneva Conventions' prohibition on moving prisoners to another country. The Israelis, citing another provision, deny any violation.

The cautious optimism in Washington that a deal may be reached soon appears to be based on the assumption here that Israel will act on its own as it begins to conclude that it risks an erosion of its crucial relationship with the United States and as American public opinion polls, such as a recent Washington Post-ABC poll, reflect rising support for some distancing in that relationship.

Reports from Israel, however, suggest that this is not a majority view at this time and that not giving in to terrorists remains a major concern. Peres, apparently reacting to that poll, said yesterday a weakening of U.S.-Israeli ties because of the hijacking would be "a surrender to the trap of terror by proxy."

Shiite Moslem leader Nabih Berri, in an interview in Beirut with CBS television, said he remains "optimistic that this can all end well" for the U.S. hostages but that "it may take two or three days" to find a way to end the stand-off. And, Berri added, he still believes Damascus is the best place for the hostages.

Berri told NBC that the hostages could be in the United States in "72 hours, and it will be exit to this affair, if the Americans help. Until now," Berri said, "I'm sorry to say that I receive help from the kidnapers but not from the United States. What we need now is help from the United States to ask the Israelis to release these people and at the same time to ask me or the kidnapers to release those people. That's all."

But the Israeli government yesterday appeared firmly committed in public to its insistence that it would not yield to terrorist demands and would only consider releasing their detainees through a direct, official request from Washington, either public or private.

In Washington, the strict blackout ordered by the administration on public discussion of the crisis continued. But Secretary of State George P. Shultz, late Wednesday night, made clear the United States is insisting that the 39 Americans be returned "immediately, unharmed and unconditionally." In a surprise move, Shultz also included seven other Americans missing for many months after having been kidnaped in Beirut.

These individuals were kidnaped by an extremist group known as Islamic Jihad, which is different from the Party of God group that hijacked the jetliner. But an official of Berri's more moderate Amal militia, which has stepped into the crisis, said his organization was ready to take Shultz's additional demand "to the quarters concerned" and that "everything can be subject to dialogue."

In contrast to the flurry of diplomatic activity touched off Wednesday in the aftermath of Berri's proposal to place the 39 hostages in the custody of the French or Swiss embassies in Beirut or in the hands of the Syrian government in Damascus while the Israelis release their prisoners, yesterday's developments appeared mostly negative.

Berri said the French "were no longer part of the negotiations." In Paris, French Foreign Minister Roland Dumas said France would not "substitute ourselves for the jailers" of the American hostages. "The liberation of the hostages must be unconditional," Dumas said, indicating that France would not accept custody of the hostages without ironclad agreement by all parties that the Israelis would release their prisoners and that France would immediately set free the Americans. Dumas had been unable to get such assurances from either Washington or Jerusalem on Wednesday.

Berri said he would take his propsal to the Swiss but was not optimistic that the hijackers would accept the Swiss terms. In Bern, the Swiss government also said that it was ready to receive the hostages but would take them only on unconditional terms and would free them.

It was after these developments that a transfer to Damascus, perhaps including a transfer to Swiss or French embassies in the Syrian capital, became even more the focus of comments by Berri in Beirut and by sources in Washington. Syria's President Hafez Assad has been a key, behind-the-scenes factor in the negotiations and has said he wants to solve the crisis quickly.

Berri said "the Syrians want them the hostages to be safe, and they want a peaceful end." Moving the hostages to Damascus would have some physical advantages for the United States, diplomatic sources said, in terms of getting them out of violence-prone Beirut, where the Shiites maintain military control, and into a country that presumably could hold them for a short period, possibly giving Israel more time to release its prisoners.

In Israel and in France yesterday, officials summed up the seeming deadlock this way: "Nobody wants to be the bad guy," as one Israeli official put it. "The French don't want to be the jailers of the hostages. The Americans don't want to ask us to yield. We don't want to yield. Everybody wants to keep clean."

"No one wants to take the responsibility for the step that has to be taken," added a French diplomat. "We have a four-sided deadlock."

On the military front, the U.S. Navy yesterday announced that a five-ship amphibious task force with 1,800 Marines aboard will sail next week for a "regularly scheduled" deployment with the U.S. 6th Fleet in the Mediterranean. But a Navy spokesman declined to say whether the group would replace or augment the three amphibious vessels now sailing between 40 and 100 miles off the Lebanese coast with four other U.S. warships.