President Reagan's top advisers yesterday endorsed a far-reaching plan to stop NASA from launching commercial and foreign satellites on the shuttle, administration sources said.

The plan, part of a new administration initiative to spur development of a private rocket industry, would effectively end the space agency's long-standing efforts to promote the shuttle as an economically self-sufficient space transportation system. Under the proposed policy, virtually all private communications satellites would be barred from future shuttle launches, leaving the shuttle to carry almost exclusively military and scientific payloads when flights resume in 1988.

Despite strong objections from the National Aeronautics and Space Administration, the plan was endorsed at a Cabinet Council meeting chaired by Treasury Secretary James A. Baker III yesterday. A White House source said that, after another meeting to iron out details, the plan will be submitted to President Reagan for inclusion in a broader administration policy statement on space.

Administration sources said yesterday there is still no consensus on providing funds for a new $2.8 billion shuttle orbiter to replace Challenger. Although Reagan has publicly endorsed a new orbiter, the project is opposed by Office of Management and Budget Director James C. Miller III, and administration officials said the president now is considering delaying any funding until the fiscal year 1988 budget is submitted to Congress next year.

The debate over how to handle commercial and foreign satellites has received less attention than a new orbiter, but administration officials and some aerospace experts say it has equally significant consequences for the future of the space program. Ever since the shuttle began flying in 1981, NASA has sold it as a "balanced" space vehicle that would launch military, scientific and commercial payloads in roughly equal proportions. In the process, NASA could recover a substantial portion of the huge national investment in the fleet from private satellite companies paying fees for launches.

The new plan, which was drafted by a task force of representatives from several departments and agencies, would bar NASA from launching most commercial and foreign payloads except for a handful of already booked satellites that were constructed specially for the shuttle. One White House source said that, at the recommendation of the State Department, it also would allow NASA to launch some foreign satellites if the president finds it necessary on national security grounds.

One administration official familiar with the plan said that of 43 NASA contracts for launching commercial and foreign payloads through 1994, only about "six or eight" would end up on the shuttle if the plan is adopted. But a White House source said the proposal will also include an option for a more gradual phasing-out of commercial customers that would permit NASA to honor a larger number of its existing contracts.

NASA has acknowledged that it would be unable to accommodate many of its commercial and foreign customers on the shuttle anyway because of a huge backlog of national security payloads. This has prompted the space agency to consider ordering additional unmanned Delta rockets to meet its commitments.

But administration officials said the proposal also would prevent NASA from launching these satellites on unmanned rockets.

One immediate consequence of the plan would be to deprive NASA of about $850 million in revenue it would receive from Western Union, American Satellite Co. and other commercial customers. These firms also have argued that barring them from U.S. subsidized shuttle flights would leave them with no choice except to turn to the French-backed Ariane rocket or the Chinese Long March rocket.

But administration officials said this is the only means to encourage U.S. aerospace companies, such as Martin Marietta Corp. and General Dynamics Corp., to enter the commercial launch business and to compete with the French, Chinese and Soviets. It also would help Transpace Carriers Inc., a Lanham, Md., firm that had won the rights to launch Delta rockets, but has been unable to attract any customers.

The drive to remove commercial customers from the shuttle also has gained momentum because of the safety issues raised in the wake of the Challenger accident. "Why should we risk somebody's life to put up an HBO satellite?" the official said.

Another element of the plan, sources said, includes guaranteeing private rocket companies' access to government-owned launch facilities, such as Cape Canaveral. It also calls for administration trade officials to seek to negotiate reductions in European Space Agency subsidies for Ariane in an effort to make U.S. rocket firms more competitive.