Declining congressional support for arms spending will force the Defense Department to choose by 1990 between a "hollow" and unprepared military or a much smaller force, and the Pentagon is likely to make the wrong choice, a panel of retired military leaders concluded yesterday.

Even if Congress grants annual increases in defense spending of 1.5 percent above the inflation rate, the panel concluded, the military will have to cut three Army divisions, six tactical fighter wings and three carrier battle groups by the end of the decade to maintain current levels of readiness. Altogether, nonnuclear forces could shrink as much as one-third.

"The more likely tendency, should defense spending be constrained, will be to retain force structure and decrease readiness," the panel said. "Under that condition, U.S. force structure of the 1990s would be approximately at the same levels of readiness and sustainability as in the late 1970s -- a hollow force."

The panel included retired admirals and generals as well as civilians who worked in the Pentagon in several administrations. The results of its year-long study, directed by retired Navy officer Harlan K. Ullman, were published yesterday by Georgetown University's Center for Strategic and International Studies.

Members of the panel agreed that reductions in the Reagan administration's five-year plan for military spending would harm national security. But the panel also agreed that such reductions appear inevitable.

In an apparent message to Defense Secretary Caspar W. Weinberger, who has lobbied unwaveringly for higher arms budgets, retired admiral Thomas H. Moorer, former chairman of the Joint Chiefs of Staff, warned against "intransigence" in the budget debate. Moorer said that the Pentagon "cannot afford to ignore fiscal or political reality, even though that reality may mean lower levels of defense spending."

The panel calculated that the Pentagon will receive $290 billion less during the next five years than it projected last January even if it manages to win 1.5 percent annual increases. If Congress grants no increases above inflation, the five-year shortfall would be $337 billion -- and if Congress holds the budget steady, without allowing for inflation, the military would fall short of projections by $516 billion.

The arms budget has increased about 50 percent above inflation since President Reagan took office. But a crunch will come even at present spending levels, the military experts said, in part because weapon costs continue to grow faster than inflation. The military must increase its procurement budget 6 percent per year above inflation just to keep pace, the panel said.

In addition, Moorer said that the 15 percent share of the budget now devoted to nuclear weapons may grow, particularly as Strategic Defense Initiative programs -- popularly known as "Star Wars" -- move from research to maturity. As a result, while many experts favor more spending on conventional weapons, "pressures for the reverse are more probable," Moorer said.

The Pentagon's likeliest response, the panel concluded, will be to save as many tanks, planes and ships as possible while letting ammunition, spare parts and other readiness accounts dwindle. In addition, Pentagon and congressional politics will dictate even cuts across the services, which the panel concluded would be the worst possible outcome.

Even the wisest decisions, however, will force choices between "two unpalatable geopolitical futures," the report said, reducing U.S. commitments in Europe or in Asia and thus undermining allies' confidence somewhere.

Robert W. Komer, undersecretary of defense in the Carter administration, offered a partial dissent to the 74-page report, suggesting that reducing duplication between services and reforming weapons buying could compensate for lower levels of military spending.