Unless the Reagan administration modifies its demand that Moscow sharply reduce its most modern and valued land-based missile forces, there is "virtually no chance" for success in either of two arms control negotiations now under way in Geneva.

That is the assessment of a 30-member panel of civilian arms control specialists and retired military officers who have been studying these issues since 1980 under auspices of the Carnegie Endowment for International Peace.

Their final report on "Challenges for U.S. National Security," to be released today, represents perhaps the most important challenge to the administration's negotiating approach to surface here.

Panel co-chairman William G. Hyland, a leading Soviet expert and former White House adviser to President Ford, told a news conference yesterday that while Moscow's approach to arms control is "troublesome," there is nothing in that approach or the Soviet record that precludes eventually reaching "an effective agreement."

But he stressed that there is "a great deal" of historical evidence that the Soviets will not accept the kind of proposals that "have been in vogue in the United States in the last two or three years."

This is a reference especially to the administration proposal at the Strategic Arms Reduction Talks (START). This would require Moscow to reduce its 2,350 land- and submarine-based missiles to 850 and would sharply restrict the land-based force, in particular, which now includes about 75 percent of Soviet nuclear warheads.

The report says "the Soviets have vigorously fought to protect" those missiles where they have "their largest investment and strategic advantage" and that these forces "are not open to major change, except incrementally over an extended period."

Despite the current gloom in relations, the report says arms control is still viable. Moscow does have economic and military incentives to try to end an American arms buildup and reach agreements in which it accepts some limitations on its land-based missile force.

But "for that to happen," the report says, "there will also have to be a considerable compromise on the American side, away from the unrealistic hopes of forcing radical reductions in and restructuring of Soviet forces."

The panel suggests that future compromises involving U.S. constraints on new cruise missiles, thousands of which are to be deployed on land, at sea and on bombers, and Soviet cutbacks on land-based missiles might be possible.

The report also opposes the kind of total nuclear freeze advocated by many anti-nuclear groups here and abroad. The panel says that halting all weapons modernization could increase the risks of unpredictable behavior on both sides in a crisis.

"Ironically, neither of the two most politically prominent proposals" these days--the freeze and Reagan's call for sweeping missile cuts--does much to increase stability in a crisis, the report says.

The report points out that cutting the number of missiles does not necessarily reduce the temptation for one side to strike first. The key is to remove the targets that are inviting to shoot at, such as big, multiple-warhead missiles on both sides.

The panel therefore strongly supports plans recently put forward by Rep. Albert Gore Jr. (D-Tenn.), former secretary of state Henry A. Kissinger and the recent presidential advisory commission headed by retired general Brent Scowcroft to develop a new small, single-warhead missile and to encourage the Soviets to go in the same direction.

The panel also contends that while the Soviets take full advantage of every loophole and imprecision in missile agreements, there is no evidence that they have violated the 1972 SALT I agreement and not yet any clear evidence that the unratified 1979 SALT II accord has been breached, although there is concern over a recent missile test.

The evidence of Soviet violations is persuasive, the report says, in chemical and biological weapons agreements. But the members in general make the case that missile treaties can be verified and that it would be difficult if not impossible for Moscow to hide a significant military development. says, in chemical and biological weapons agreements. But the members in general make the case that missile treaties can be verified and that it would be difficult if not impossible for Moscow to hide a significant military development.