President Reagan warned the hijackers of a Trans World Airlines jet yesterday to release their remaining hostages "for their own safety" and hinted at retaliation if any of them are harmed.

However, Reagan and other administration officials talked optimistically about the possibility of a negotiated solution to the Mideast hijacking crisis that diplomatic sources said could involve freeing the American passengers in exchange for the future release of hundreds of Shiite militiamen held by Israel.

Speaking of the hijackers, an Arab diplomat close to the situation said, "There is a good chance they will accept a deal."

At the same time, an administration official said there appeared to be "no viable military option" for retaliating against groups that may have sponsored the hijacking. Reagan's warning was directed at the hijackers themselves, the official said.

Reagan, who had deliberately played a low-profile role during the first two days of the crisis, cut short his weekend stay at Camp David to return to Washington late yesterday morning and take charge of the American response.

Instead of brushing past reporters without comment as he had done when he departed for Camp David, the president stopped on the White House lawn to answer questions and say that the U.S. government was talking to the governments of Lebanon, Syria and Israel in an effort to resolve the crisis peacefully. He then went into a 75-minute meeting on the crisis with his top aides.

Later, White House spokesman Larry Speakes issued a statement saying that the president remained "hopeful for an early peaceful resolution" of the incident and quoting him as saying, "I will not consider the matter resolved until all passengers and the crew are safe."

Speakes said "all passengers" included the six to 10 persons who were taken off the plane early Saturday by members of the hijacking team and are thought to be held somewhere in Beirut.

Both the president and his spokesman reiterated the administration's formal opposition to negotiating with terrorists.

"The U.S. government policy remains the same," Speakes said. "We do not make concessions, we do not give in to demands and we do not encourage other governments to do it."

Despite these words, administration and diplomatic sources acknowledged discussions with the hijackers about turning loose the passengers in exchange for an eventual release of some or all of the Shiite prisoners held by the Israelis. These sources said that both the International Committee of the Red Cross and the Algerian government were playing leading roles in these discussions.

Reagan sent another message yesterday to Algerian President Chadli Bendjedid asking for his assistance. Administration officials have repeatedly expressed gratitude for the efforts of the Algerians, and it was the latest departure of the TWA plane yesterday morning from Algiers to Beirut that apparently convinced the president to return to the White House.

"I decided with this third trip to Beirut now and the activity that is going on that I'd rather be here face to face than dealing on the telephone," Reagan said.

Deputy press secretary Robert Sims said that Reagan, while still in Camp David, had talked half a dozen times about the hijacking Saturday night and yesterday morning with national security affairs adviser Robert C. McFarlane. The president canceled his appearance at a picnic sponsored yesterday by the Big Brothers and returned by helicopter to the White House.

Reagan has always opposed negotiation with terrorists, and Secretary of State George P. Shultz and McFarlane have warned on several occasions that the United States reserves the right to retaliate against terrorism. Yesterday, the nuclear-powered aircraft carrier USS Nimitz, which had been scheduled to put into port in Italy, changed course and headed for the eastern Mediterranean.

But despite this symbolic action, the administration appeared to be embarked on a course of negotiation rather than retaliation. Officials repeatedly stressed the role of the Red Cross in what they preferred to call "discussions" rather than "negotiations."

While the U.S. government welcomed the activities of the Red Cross and the Algerian government, administration officials said the United States would not go so far as to ask the Israelis to release the Shiite prisoners.

Administration officials, deeply pessimistic when the plane flew back to Beirut, began showing signs of optimism yesterday. One manifestation of this was Reagan's willingness to take a high-profile position after two days of making virtually no comments about the situation.

Wearing a plaid shirt and appearing somewhat tired, Reagan returned to the White House at 11:15 a.m. and stopped on the South Lawn to take questions. The helicopter engines were turned off, a reliable sign in the Reagan White House that television coverage was desired.

Reagan said he was "very much encouraged" by "the very fact that it has gone on this long without any . . . general destruction and massacre." He said of the hijackers that he hoped "that they themselves will see that, for their own safety, that they'd better turn those people loose."

Asked what he was suggesting by this remark, Reagan said, "Well, there have been instances in which hijackers have found that action was taken that resulted in their death or capture . . . . "