A phantom hijacker held a TWA jet captive for 11 hours yesterday while airline officials tried to decide how to deal with his bizarre threat to blow up the plane with all 87 persons aboard.

Finally, mystified TWA officials decided to evacute the plane, and the 78 passengers and nine crew members - with the unknown hijacker apparently in their midst - filed off safely.

Swiss police, after questioning all 87 persons for hours, admitted last night that they had not found the hijacker and were not sure whether the whole episode was a hoax or a serious terrorist attempt.

"Up to now, we cannot rule out either," said Swiss Justice Minister Kurt Furgler. Added TWA Vice President Stewart Long: "It was the work of a madman."

The bizarre drama began when a 19-page list of demands was handed to a stewardess in a passenger cabin darkened for a movie as the Boeing 707, en route from New York, was flying over the Irish coast. It flew another 900 miles and landed in Geneva, its original destination.

In the document, the "task force of revolutionary soldiers" threatened to blow up the plane. It said two suitcases filled with explosives were in the baggage compartment and would be detonated by remote control at 5:30 p.m. if its demands were not met.

The "task force" demanded freedom for former Nazi deputy fuehrer Rudolf Hess, jailed in Spandau Prison in Berlin; for Sirhan Sirhan, convicted assassin of Robert F. Kennedy and for five Croatian hijackers also jailed in the United States.

According to police, passengers said the document was handed to the stewardess by a man who appeared to be wearing a wig. Some witnesses also said the hijacker was wearing a false beard.

The stewardess then took the 19-page document to the pilot. Capt. Robert Hamilton who radioed ahead that he was proceeding as instructed to Geneva "under control by elements."

The plane landed at Cointrin Airport here and parked about 300 yards from the terminal building. The pilot, who remained locked in the cockpit, said the plane was being held by hijackers, and the jetliner was surrounded by a loose cordon of police.

The Swiss government hastily assembled a crisis staff and kept in touch with U.S. authorities. Police, meanwhile, pored over the 19-page document which had been thrown from the cockpit of the plane, trying to decide what action to take.

As the hours dragged by, passengers read, slept, or suspiciously eyed each other.

"While we were waiting, passengers began to look at each other trying to figure out who was the hijacker," said passenger Nina Axelrod of New York.

When the deadline for the detonation of the alleged bombs neared, the pilot finally was instructed to announce over the plane's intercom that two "plenipotentiaries" were coming aboard to negotiate the release of the women and children hostages.

"There was no reaction so the two boarded the plane," one official said.

TWA Vice President Long, one of those who boarded the jetliner, said he "walked up and down the aisle of the plane several times offering to talk to the hijacker.

"No one talked to me and I thought, 'Let's get off this plane,'" Long said. So the pilot instructed the passengers to leave, and they hurried down landing gangways.

Two blue buses quickly took the passengers and crew to the airport terminal, where they were finger-printed and questioned by authorities.

"Obviously, the person that drew up the letter walked off with the other passengers," said a TWA spokesman in New York.

But after three hours of questioning those aboard the airliner, Geneva Security Police Chief Roger Warinsky said: "We have not found any suspect."

The passengers were subsequently taken to Geneva hotels to spend the night. Among those aboard the plane was Katarina van den Heuvel, 18-year-old daughter of William van den Heuvel, the U.S. ambassador to the United Nations offices here, and Chester Davenport, assistant secretary of transportion for policy and international affairs.

Warinsky later said that a search of the plane failed to turn up any weapons or the wig or beard the phantom hijacker was variously reported to have worn.

Federal Aviation Administration officials in Washington said it was "pretty clear" that contrary to what the list of demands said, there were no explosives in the baggage compartment of the jetliner. The FAA said U.S. airport security checkpoints would have detected explosives.

The plane, flight 83, had departed New York Thursday night. It included passengers from a flight that began in Tulsa, Okla, and had made intermediate stops in St. Louis and Detroit. TWA said the plane was to have flown on to Nice, France, after a stop in Geneva.

Swiss Justice Ministry spokesman Ulrich Hubacher said the list of demands purported to be from a group called "United Revolutionary Soldiers of the Reciprocal Relief Alliance for Peace, Justice and Freedom Everywhere."

An FBI official in Washington and Swiss intelligence officials said they had never heard of such a group.

Officials said the demands called for release of five Croatians imprisoned in the United States for the hijacking of a TWA airliner from New York in September 1976. The five, who left behind a bomb that killed a New York City policeman, were apprehended after the plane landed in Paris.

The list demanded that Hess, 84, be joined by his wife, son and lawyer that their physician, identified as Dr. E. Bucher, act as go-between; and that they all be at the Geneva airport at 5 p.m. It also demanded that President Carter publicly announce that Hess had been released.