Administration and congressional sources expressed hope yesterday that the Pentagon's portion of the space program will not be set back seriously by the explosion of the space shuttle Challenger unless the accident investigation determines that fundamental changes are required in the remaining three shuttles.

Problems could develop, however, if a new space orbiter is not built to replace Challenger in time to meet the Pentagon's ambitious schedule for 10 or more annual satellite launches beginning in 1988, sources said.

The Defense Department is the best paying customer for the National Aeronautics and Space Administration shuttles. The Pentagon uses the space vehicle's substantial carrying capacity to launch large intelligence, communications and weather satellites and conduct experiments for President Reagan's Strategic Defense Initiative, the so-called "Star Wars" research program.

The Pentagon also puts smaller military satellites into orbit using older booster rockets that once were part of the intercontinental ballistic missile program.

In February 1985, Reagan approved an agreement between the two agencies that guaranteed the Pentagon would take one-third of the planned 24 shuttle flights the government hopes to send up annually by 1988.

That made NASA's shuttles the primary launch vehicles for the Pentagon's $8 billion worth of satellites scheduled to go into orbit in each of the next five years, according to data given to Congress.

The Pentagon, long concerned that NASA's four shuttles would not be available to carry all of the planned military satellites into orbit on the schedule the military believed necessary, persuaded Congress to approve modification of old Titan missile boosters as they were taken out of their silos and deactivated as nuclear weapon launchers.

In addition, the Pentagon received approval for production of 10 giant new launching rockets of its own.

These so-called complementary expendable launch vehicles (CELV) will not to be available before 1988. Together with the modified Titans, they would permit the Pentagon to launch up to four satellites a year independent of the shuttle, payloads that the military did not want subject to shuttle delays.

Only four military shuttle flights are scheduled in 1986, with the first set for July, according to an Air Force spokesman. That shuttle would carry an experiment called Teal Ruby, designed to test the ability to track bombers and other aircraft from space.

Teal Ruby is already more than a year behind schedule and another delay caused by the shuttle mishap, would not affect the program, sources said.

A more important military shuttle launch is set for September at Vandenberg Air Force Base, Calif., when the Air Force is to place in orbit its newest photographic intelligence satellite, the KH12. This device must travel into space by shuttle since it is too large to be launched by the older booster rockets.

An earlier model of the satellite, the smaller KH11, was destroyed last August when the unmanned Air Force booster rocket carrying it into orbit blew up.

Another shuttle flight toward the end of this year is expected to take a military communications satellite into space, sources said.

The Pentagon and NASA have carried on a low-keyed battle the past few years over who would control the future launching capability for the military space program.

That program is to expand over the next decade as new multisatellite navigation and communication systems are put into space, and preparations are made for Star Wars experiments in space.

As NASA developed the shuttle program, it looked to the Pentagon for financial support -- the Pentagon pays $55 million for each flight it uses -- and for stability in long-term use.

The president's order last February guaranteeing Pentagon support for one-third of the shuttle flights was considered only a temporary solution. A NASA-Defense Department study is under way to determine whether the shuttle or another unmanned space launch vehicle should be developed to handle military satellites after 1995.

NASA's Jesse W. Moore, associate administrator for space flight, told Congress last year that each shuttle orbiter would last for 100 flights or more, taking them well beyond the year 2000.

Pentagon officials are known to lean toward building yet another unmanned launch vehicle that would be available for military crises as well as times when shuttles are not available.

In congressional testimony last year, Air Force Undersecretary Edward C. Aldridge said the Pentagon worried that "in the future the shuttles could be grounded over some short period of time . . . or one of them could be damaged to the point it had to be pulled out of service."

In the years after 1991, the Pentagon plans to put eight of payloads annually in NASA shuttles, while another two or three launches in each of those years would be done by either Titan II or CELV boosters.

The present unmanned Air Force booster rockets can carry satellites that weigh no more than 3,000 pounds. The shuttle can carry a total load up to 32,000 pounds. The largest military satellites weigh up to 10,000 pounds.

The modified Titan boosters will carry 3,000-pound loads while the new rocket boosters approved by Congress will be able to lift 10,000-pound satellites.

Some NASA officials opposed past Pentagon plans to continue producing unmanned boosters for its smaller satellites on grounds that those launches could be handled by a fifth shuttle orbiter. Now administration and Capitol Hill sources say the fifth shuttle will have to be built to replace the one destroyed yesterday.

A Pentagon source added that the tragedy confirmed fears of shuttle availabilities and guaranteed that a new booster system would be developed to handle launches in the next decade.