Negotiations over the fate of more than 30 hostages still aboard Trans World Airlines Flight 847 intensified yesterday after the extremist Shiite Moslem hijackers forced the plane to return once more to Beirut from Algiers.

The hijackers, now speaking through the established Shiite leadership in Beirut, were sticking to their fundamental demand, that Israel release between 700 and 800 Shiite Moslems taken prisoner during the Israeli occupation of Lebanon and held in an Israeli prison near Haifa.

The Israeli Cabinet met in Jerusalem to discuss those prisoners. Before the hijacking, Israel had said it planned to release them as it withdrew from southern Lebanon. Israeli officials have said they would not negotiate with the hijackers but said today they would consider a formal U.S. request to swap the prisoners for the passengers on Flight 847. They added that no request had been received.

In Washington, U.S. officials said no such request would be made, consistent with American policy not to give in to demands from terrorists.

However, both diplomatic and U.S. sources indicated that there are other possibilities for a negotiated settlement to the standoff, perhaps through the International Committee of the Red Cross, whose representatives were involved in talks in Beirut and Algiers.

Arab diplomatic sources expressed optimism that the hijackers might release the passengers if they were confident that Israel would release its Shiite prisoners soon afterward. There were strong indications last night that much of the diplomatic activity aimed at resolving the hostage crisis is going on out of public view.

The White House yesterday abandoned the low- to no-profile approach it had taken to the hijacking. President Reagan cut short his Camp David weekend in late morning, and went before television cameras on the White House lawn to warn the hijackers that they should release the hostages "for their own safety."

In Beirut, where the plane landed for the third time in as many days, a note to President Reagan urging him to refrain from using force to bring the crisis to an end was issued, purportedly by the passengers still aboard.

"We the undersigned 32 American hostages aboard Flight 847 are writing you freely, not under duress," the note said. "We implore you not to take any direct military action on our behalf."

The first of the hostages released Saturday in Algiers returned to the United States yesterday. Several survivors of the ordeal reported rough treatment at the hands of their captors. An ailing passenger freed by the hijackers yesterday described conditions aboard the plane as "hell."

Shiite Amal leader Nabih Berri, Lebanon's justice minister, assumed negotiations on behalf of the hijackers at their request, an indication of the Shiites' increasing role in the Middle East in general and Lebanon in particular. Berri reported that the hijackers had promised not to harm anyone if their demands were met.

The hijackers also demanded to see the ambassadors of Britain, France and Spain and representatives of the Red Cross and the United Nations, and Berri said he had held such meetings for the hijackers. U.S. Ambassador to Lebanon Reginald Bartholomew was also in contact with Berri yesterday.

Throughout the ordeal, the question of just how many hostages there have been at any given time has been troublesome. The picture was clouded further yesterday when reports surfaced that some of the passengers -- Americans with Jewish-sounding names -- had been whisked off the plane in the dark during the second stop in Beirut early Saturday morning. A flight attendant freed by the hijackers said she had been ordered to select those passengers. She said she had refused but that she had handed over all the passengers' passports.

The attendant said that there had been four marines aboard and that she believed the unidentified passenger killed Saturday by the hijackers was one of them. A released passenger, however, said the victim was a member of a U.S. Navy diving team.

"We are aware that some six to 10 were removed in Beirut and led away and presumably are being held somewhere in Beirut," a White House spokesman said. "We are making every effort through Lebanese authorities to . . . secure their release." The White House said the passengers taken off the plane would have to be freed as part of any resolution of the crisis.

Late last night journalists were ordered away from the Beirut airport as Shiite militiamen, fearing a commando strike to free the hostages, went on alert. Earlier, Moslem gunners on the coast nearby opened fire on a vessel believed to be a warship. There were reports that a guided-missile destroyer, the USS Kidd, had left Haifa, Israel, a week earlier than scheduled and that the aircraft carrier USS Nimitz was heading for the eastern Mediterranean.