The colony of carpenter ants and their queen that flew aboard the space shuttle Challenger last month to see how their social structure would survive the rigors of space flight failed to clear a major hurdle. They were apparently dead on their return to Earth.

While nobody will be certain that all 151 ants are dead until the casing housing the colony is opened, no signs of life in the normally bustling colony were visible Tuesday through the clear plastic top.

"There was no motion at all inside the colony," said Nicholas Timpanelli, the teacher who accompanied students from Woodrow Wilson and Camden high schools in New Jersey to Cape Canaveral to recover their experiment. "I counted six dead ants, which suggests they're all dead."

The canister that housed the colony in space was to be flown yesterday to Camden. It will be opened either today or Friday in the presence of two biology professors from Philadelphia's Temple University who served as advisers to the students who designed the experiment in 1981. The students were supported by a $10,000 grant from RCA Corp. to construct the experiment.

"The kids are sorry the ants are dead," Timpanelli said yesterday, "but they're very happy with their experiment. They did something. They put an experiment into space that worked."

Timpanelli said the first thing he and the two students did when they got to Cape Canaveral to check on results of their experiment was to see if the electrical connections and batteries that supplied heat and cooling to the colony were still working. They were.

So were the lights that were tripped on by computer command from the shuttle's cabin, and so were the two cameras that photographed the colony. One camera took five seconds of film every 15 minutes, the other made a television tape recording every seven hours the ant colony was in flight.

"We turned on the experiment Tuesday and we could hear the fans and the movie camera running," Timpanelli said. "We could also hear the clicking of the other camera and we could hear the video tape drive running, so we know there was nothing wrong with the experiment."

Also tested on Tuesday was the device that turned on the experiment when Challenger left Earth and reached an altitude of 5,000 feet. Timpanelli said that it worked, too, which meant that death did not come to the ant colony for want of heat and cooling when they needed it.

What caused their death? The films may reveal in detail how the ants died if they died in space. The trouble is, the ants had been confined inside their canister in Challenger's cargo bay since April 24, almost two months before Challenger's launching. They could have died before Challenger took off, though it's unlikely, because two control colonies of the same number of carpenter ants survived back at the high schools in Camden.

There is a chance that the ants died in the heat of the California desert, where they were cooped up for eight days after the shuttle made its unscheduled landing at Edwards Air Force Base when bad weather diverted it from landing at Florida's Kennedy Space Center. In any event, the film will show when the ants died, though not necessarily how.

Carpenter ants were chosen to be the first ant colony in space because, like bees, they have a queen and a social structure that can be observed. They have hard external skeletons to stand up to space flight, they live in confined spaces that require little support, they have feet that can cling to smooth surfaces without the help of gravity and they tend to keep to themselves and not wander off like regular ants.