The National Aeronautics and Space Administration last night canceled the next mission of the space shuttle Challenger because of difficulties with electrical systems in a $100 million communications satellite Challenger was to carry into orbit Thursday.

NASA said the 100-ton spaceliner will be hauled from its launch pad to the Vehicle Assembly Building, where the Tracking and Data Relay Satellite (TDRS) will be removed and overhauled.

Canada's Telesat satellite also will be removed from Challenger's cargo bay and placed in the cargo bay of the space shuttle Discovery, which is to carry a U.S. Syncom satellite into orbit later this month.

The cancellation delays indefinitely the flight of the TDRS, the second of three such satellites that are to form a radio relay network to serve NASA for the next 20 years.

It also delays indefinitely the space flight of Sen. Jake Garn (R-Utah), who was to become the first civilian non-scientist to fly in space.

Challenger's mission had been delayed three times from the original Feb. 7 launch date for small difficulties encountered on the launch pad. The latest difficulties were more serious, involving the failure of one of 24 electrical cells that feed power through the TDRS satellite.

Two fresh problems cropped up yesterday, one involving an erratic timing device that could have triggered major transmission and receiving errors once the satellite reached orbit. And signals received from a coded device used to screen out stray powerful radar signals from the Soviet Union suggested further trouble, leading NASA officials to cancel the flight.

NASA said it has no idea when the Challenger flight will be rescheduled, noting that the entire shuttle manifest for 1985 will have to be juggled to accommodate all of its customers.