A $100 million satellite launched on the maiden flight of the Challenger space shuttle tumbled wildly through space today until it was stabilized just before it would have spun out of control forever.

In an intense, three-hour struggle, ground-based flight directors from the Air Force and the civilian space agency stabilized the flight of the 5,000-pound communications satellite by beaming a continuous signal--"separate, separate"--into space to direct the satellite to shed its misfiring engine.

At 9 a.m. (EST) today, the tumbling suddenly stopped and the satellite began sending signals to Earth. It remained in an erratic orbit that threatened its effectiveness as a communications relay station for future shuttle flights and other purposes, but flight directors were hoping to correct the orbit in the next two days.

The incident nearly became one of the most costly mishaps in the 25 years since the United States began flying in space.

It began when the communications satellite, called the Tracking Data and Relay Satellite (TDRS), was deployed smoothly this morning from Challenger by astronauts Paul J. Weitz, Karol J. Bobko, Story Musgrave and Donald H. Peterson.

"At one point this morning we were concerned that we were going to lose the satellite," said Robert O. Aller, director of NASA's $3 billion TDRS program. "We got it back under control only because of the outstanding efforts of an excellent flight control team that reminds me that this business is one of men and machines, not just machines."

The tracking satellite is the first of three that will be deployed in the next 11 months to serve as the communications command post for every civilian and military satellite the United States puts into Earth orbit for the next 20 years.

The Landsat, or mapping, satellite now in orbit, the Spacelab due to fly on the ninth flight of the space shuttle and the Space Telescope scheduled to be orbited in 1985 all would be useless without the tracking satellite.

Forced into the wild tumble by an apparent misfire of an onboard rocket engine, the tracking satellite has been described as the most complex and costly ever built. It was lost in space for three hours as antennas all around the world tried to find it. Even as the satellite was tumbling out of control, it was also losing the battery power that would allow it to respond to radio commands from Earth should the signals reach it.

Sources here described the scene this morning, in engineering rooms scattered across the campus-like expanse of the Johnson Space Center, as one of chaos. "We've lost it," one source said a senior NASA official cried out when the satellite could not be found after two hours of searching. "Forget it," another said a little earlier. "Wake up Beggs and lock it up for the investigation."

Beggs is James M. Beggs, NASA administrator, and the "investigation" is what could be expected to follow. In fact, even though the satellite was brought under control, Air Force and NASA officials announced later today that the matter would be investigated.

Almost forgotten in the midst of events were Challenger and the activities of its four-man crew. Musgrave, a surgeon as well as an astronaut, tested ways to make extra pure medicine in the weightlessness of space. Weitz and Bobko practiced maneuvers for an orbital repair mission scheduled for 1984 and Peterson manned the TV camera for a visual tour of the Challenger's cabin.

Earlier, just after midnight, astronauts Musgrave and Peterson had deployed the satellite out of Challenger's cargo bay, and astronauts Weitz and Bobko backed Challenger away to a safe distance of 32 miles to await the first of two engine burns that would take the satellite to a permanent perch in a much higher orbit than Challenger.

The first engine burn was perfect. The second burn was scheduled for 6 a.m., when a signal was sent from Earth to fire the satellite's onboard engine to place it in orbit 22,400 miles above the Earth over the equator just off the coast of Brazil. There, the satellite was intended to match the Earth's rotational speed and stay in place for the next 10 years.

The satellite must have received the signal, because the engine fired for at least 70 seconds of a planned 104-second burn. Then everything stopped. There was no more radio signal from the satellite. It would no longer respond to commands and it began to tumble at what NASA's Aller called a "high rate of rotation."

Tracking antennas at the Air Force Satellite Control Facility in Sunnyvale and at NASA's Goldstone in California, on Merritt Island in Florida, on Ascension Island in the South Atlantic and on the islands of Guam and Hawaii in the Pacific Ocean were ordered to try to contact the satellite.

For almost three hours, the satellite flew through space deaf to all commands. Suddenly, just after 9 a.m., the antennas at Goldstone in California's Mojave Desert received a signal that the satellite had separated from its engine.

Soon, other antennas heard that the satellite had unfolded its solar panels, which would enable it to stop using battery power.

Flight directors had estimated that the satellite's batteries were about to expire, which would prevent it from communicating with Earth, even if ground controllers could send messages to it

Getting the satellite back under control, however, did not rescue its mission. The apparent misfiring of the engine has put the satellite in an erratic orbit that is 21,950 miles above the Earth at its high point and 13,540 miles above the Earth at its low point. The satellite is also on a path that crosses the equator at an angle of 2.4 degrees, instead of flying almost directly over the equator.

Flight directors today were discussing ways to get the satellite back in its proper orbit. They were considering whether to use up "several hundred pounds" of the 1,300 pounds of hydrazine fuel on board the satellite to move it into a better orbit. The hydrazine was placed on the satellite to fuel a small engine that would be fired periodically to keep the satellite from drifting out of orbit.

"We think we can get pretty close to what would have been our nominal orbit," Aller said. "We certainly have more fuel than we need on board to operate the satellite. The question is how to use that fuel in the best way to recover as much as we can of what we've lost."

"We're working toward putting it where it belongs as soon as we can," said Ed Smylie, NASA's associate administrator for tracking and data systems.

The time it might take to put the satellite into a circular orbit is "measured in days to weeks," he said. "We're likely to do it in series of burns, over a period of time."

He said officials "believe we have the ability to reach the desired orbit and we will proceed to do that over the next several days to two weeks."

A decision on how and when to start the rescue sequence will probably be made late Wednesday, he said.

"There is no hurry," he said. "The spacecraft is in a safe condition . . . . We feel no great pressure to make this decision in advance of having all the data in hand that we feel we need."