A tight-lipped Reagan administration once again put its hopes on diplomacy yesterday amid a swirl of confidential messages involving France, Israel, Syria and the Lebanese Shiites who are holding 39 American hostages from a hijacked TWA airliner.

One day after President Reagan set a deadline of "several days" for diplomacy to work before ordering punitive measures, the administration dropped all talk of retaliation and clamped a lid on information about the fast-developing diplomatic track. While cautioning that problems remained to be resolved, officials were notably more hopeful about an early solution than they had been since the first weekend of the 13-day hijack-hostage drama.

"It is a very important stage, and it's very hopeful," said Senate Foreign Relations Committee Chairman Richard G. Lugar (R-Ind.) after a meeting at the White House. He refused to be specific.

The two main lines of U.S. diplomatic activity, according to official sources, are efforts to persuade Lebanese Shiite leader Nabih Berri and the more radical Hezbollah faction to release the 39 remaining American hostages unconditionally, and a parallel effort with France, Syria and Israel to transfer the Americans to the temporary custody of a third party pending Israel's release of more than 700 Lebanese prisoners, mostly Shiites.

A major sticking point was Washington's insistence that the release of the U.S. hostages not be formally linked to Israel's freeing of its prisoners, as demanded by Berri. The White House repeated yesterday that "we will not ask Israel" to release additional Lebanese prisoners, and State Department sources said the U.S. stand still rejected "a linked release." Only some kind of "inexplicit, indirect" linkage is acceptable, one official said.

Another uncertainty for U.S. officials was whether Berri could deliver all 39 hostages to a third party before an actual release of the Israeli-held prisoners. Some officials said they were not sure that Berri could produce all the Americans, some of whom are held by the more radical Hezbollah faction, even if the Israelis go through with a full prisoner release.

White House spokesman Larry Speakes refused to comment on Berri's statement in Beirut raising the possibility of transfer of the hostages to a third party. "Our position is that all of the hostages should be released immediately, and without preconditions," Speakes said.

Speakes acknowledged that the administration has been "in close touch" with the Syrian government of President Hafez Assad in an effort to resolve the situation. The spokesman, as in the past, said Assad "expressed a desire to be helpful and is in a position to be helpful." But unlike a statement on the same subject Tuesday, Speakes did not add this time that there is "no tangible evidence" that Syria is actually helping. State Department sources said reports reaching Washington indicate that Berri visited the Syrian capital of Damascus in an unannounced trip earlier in the week. This was taken as a sign of increasing Syrian diplomatic activity aimed at freeing the hostages.

Administration officials pointed to two developments of the past day as the most clear-cut signs of movement in a positive direction.

One was the visit by representatives of the International Red Cross to each of the U.S. hostages. Despite the fact that all the Americans except the TWA crew were seen by the Red Cross in one place, U.S. officials said they still believe that the hostages are being held in several places and by representatives of the two Lebanese Shiite factions -- Amal and Hezbollah.

Berri's ability to bring all the passenger hostages together was the first tangible display of his authority over those hostages in custody of Hezbollah, a fundamentalist pro-Iranian group.

The other positive development was the change in Berri's attitude as expressed in his public statements. For the first time, officials noted, he was placing more emphasis on getting the hostages out of his hands than on demands being made of the United States or Israel.

White House officials said the administration was not aware of Berri's proposal for transfer of the hostages when Reagan's deadline was announced late Tuesday.

"Berri is on the hot seat and he is trying to get off," an official said. He explained that Berri initially assumed overall responsibility for all U.S. hostages under the explicit agreement with Hezbollah that the Americans would be freed only in return for release of the more than 700 Lebanese held in Israel. But after making this deal, Berri found that the United States adamantly opposed such linkage, and Israel insisted on a U.S. go-ahead for its part of the arrangement.

As part of its diplomatic effort, the United States has exchanged messages in recent days with the government of Iran, officials said. But the upshot, as interpreted in Washington, is that "Iran won't help." A statement by Ali Akbar Hashemi Rafsanjani, Iran's Majlis [parliament] speaker, during a visit to Damascus Monday that "in general, we do not approve of any acts that terrorize innocent people" was cited by the U.S. officials as the limit of Iran's helpfulness. Rafsanjani met during his Damascus trip with leaders of Hezbollah.

U.S. officials with memories of the 1979-'81 travail of the U.S. diplomatic hostages held in Iran were particularly cautious about yesterday's developments. In the earlier situation, the Carter administration pinned much hope on a transfer of the American hostages from the control of radical students to the Iranian government. It seemed tantalizingly close at times, including the day of the Wisconsin presidential primary when President Jimmy Carter expressed optimism in a celebrated dawn press conference. But the transfer never took place.