The five Challenger astronauts went to sleep early tonight in preparation for a dramatic effort Wednesday to ease a satellite out into space, fly in tandem with it for eight hours and then pull it back aboard their space shuttle with a huge mechanical arm--the forerunner of future satellite repair missions.

"We believe we're going to do something we've never done before in this program," flight director John Cox said at the Johnson Space Center in Houston. "That is, deploying and retrieving a satellite." The satellite is a 3,200-pound instrument package built by the West German firm of Messerschmitt-Bolkow-Blohm.

Early Wednesday morning, astronauts Sally K. Ride and John M. Fabian will manipulate the shuttle's 50-foot arm to raise the satellite from Challenger's cargo bay and drop it off into space.

Astronauts Robert L. Crippen and Frederick H. Hauck then will maneuver Challenger to a position about 1,000 feet away from the satellite, maintain the separation for almost eight hours, then rendezvous with the satellite so Ride and Fabian can retrieve it with the arm and stow it back in the cargo bay.

The seventh flight of the space shuttle and the first with a woman astronaut aboard was still going "swimmingly," except for minor problems regarded by flight directors as little more than a nuisance.

The eight instruments aboard the Shuttle Pallet Satellite were overheating somewhat, but flight directors insisted that none of the data was being lost and none of the instruments was damaged.

Twice the astronauts sent back to Earth telecasts depicting their activities inside the cabin and their preparations for Wednesday's deployment and retrieval of the German satellite.

Ride was the tour guide for the first telecast, which showed her tending to an experiment that separates three proteins in blood by taking advantage of the different electrostatic charge on the surface of each.

"Isn't science wonderful?" Ride said. "Actually we're not going to be seeing much here at all. We don't even get the first part of the sample going through for another 30 minutes.

"But I can tell you that the first sample is an all-American sample. It's got three color dyes in it, red, white and blue."

Later the astronauts showed off the three-foot-wide antenna that will maintain radar contact with the West German satellite when it moves away from the shuttle.

The same antenna, the largest ever flown on any manned spacecraft, will be used on later flights to communicate with Earth through a $100 million communications satellite deployed by the sixth shuttle crew last April but only now nearing its proper orbit.

The crew held up a bag of jelly beans given to them by President Reagan during a recent visit to the White House. Astronaut Norman E. Thagard squeezed the bag gently, and the multi-colored jelly beans popped out of the bag and floated around the cabin as the crew tapped them to and fro and caught them in their mouths.

The five astronauts are due to return to Earth just before 7 a.m. Friday. Concern that the weather might postpone the landing a day or two began to dissipate late today with a forecast that heavy clouds and thunderstorms that covered Cape Canaveral most of today would disappear by Wednesday.