A high-powered group of American arms experts, including two Democratic former secretaries of defense, two key Republican senators and former secretary of state Alexander M. Haig Jr., yesterday published a report that concludes that the Reagan administration's "Star Wars" program appears unrealistic and urges that it be scaled down to pursuit of a more limited defense system.

"Despite advances in technology, a ballistic missile defense that could protect American and allied populations with tolerably low leak rates does not now appear to be a realistic possibility," according to the Georgetown Center for Strategic and International Studies.

It seems "illusory to expect that we can escape the condition of mutual vulnerability that has prevailed through much of the nuclear age," the report says. This appeared to be an attack on President Reagan's assertions that his Strategic Defense Initiative holds out the hope of doing away with the doctine of Mutual Assured Destruction (MAD), long the basis of U.S. and Soviet deterrence thinking.

While going ahead with research on a "broad technological front," the United States should focus instead in the near term on protecting its missile forces as well as its command, control and communications facilities, the report said.

The report by the 32-member panel also concluded that the arms race is a symptom, not the cause, of an "intense global rivalry" between the United States and the Soviet Union that can be successfully managed to reduce the risk of nuclear war despite the profound differences and a long-term competitive relationship.

Two well-known supporters of the administration's Strategic Defense Initiative, Sen. John W. Warner (R-Va.), chairman of the Armed Services subcommittee on strategic and theater nuclear forces, and Sen. Jake Garn (R-Utah.), a member of the Appropriations defense subcommittee, disassociated themselves from the "Star Wars" conclusions.

Altogether, nine members entered dissenting opinions on various of the report's conclusions, while six others, including former secretary of state Henry A. Kissinger, Rep. Les Aspin (D-Wis.), chairman of the House Armed Services Committee, and three top administration officials did not associate themselves with the report at all.

Haig joined Eugene V. Rostow, the administration's first director of the Arms Control and Disarmament Agency, in a nine-page "statement of partial dissent" about the study's assessment of overall Soviet behavior and intentions and the possibility of reaching any real detente with Moscow because of its "expansionism."

Haig did not specifically disassociate himself, however, from its judgment about the feasibility of the Star Wars space-based defense or another conclusion that "the restoration of clear-cut American military superiority is, for all practical purposes, beyond our reach."

Called "Reducing the Risk of Nuclear War," the 64-page document is the first of a series of studies by private bipartisan groups seeking to establish some consensus about problems posed by the Soviet arms buildup for U.S. deterrence and arms control policies.

The number of dissenters seemed only to highlight, however, the extreme difficulty of getting agreement among experts and policy-makers on the complicated issues of arms control, strategic defense and, above all, the president's Strategic Defense Initiative.

The report argues that the United States, in seeking to improve the credibility of its deterrence and promote greater stability, should adopt a policy of what it calls "unilateral defense programs."

These, it said, should include ensuring the survival of U.S. strategic forces through a more effective defensive system but also through the deployment of mobile and fixed Midgetman intercontinental ballistic missiles in the 1990s, more Trident submarines, cruise missiles and bombers and research into a new, smaller nuclear-powered ballistic missile submarine.

The report took no position on the controversial MX missile.