The Gramm-Rudman-Hollings budget-balancing act will dramatically reverse President Reagan's military buildup and lead to "the greatest defense cutback in peacetime history," military and budget committee leaders in Congress have concluded.

In separate analyses, Senate Republican and House Democratic leaders predict that the landmark budget law will force cuts of $65 billion to $90 billion in defense spending authority in fiscal 1987, after an estimated cut of $10 billion to $13 billion for the current fiscal year.

Instead of growing at a rate of 3 percent after inflation, as planned last year in a White House-congressional compromise, new spending authority for defense would decline from its current level of almost $300 billion to about $260 billion, or perhaps less, the legislators said.

In a study released today, House Armed Services Committee Chairman Les Aspin (D-Wis.) said a cut of $65 billion from inflation-adjusted current programs will be required for fiscal 1987, resulting in a 20 percent shortfall from Reagan's defense-spending target.

"This isn't just tinkering at the edges of the defense budget. It means we are reversing the defense buildup of the last few years. We are marching down the mountain we have been marching up," Aspin said.

"[Defense Secretary] Caspar Weinberger, who has prided himself on presiding over the greatest defense buildup in peacetime history, is about to preside over the greatest defense cutback in peacetime history," he added.

Senate Budget Committee Chairman Pete V. Domenici (R-N.M.), in an interview last week, predicted deeper cuts in defense spending authority -- about $90 billion in fiscal 1987.

"It is not livable, but it is likely," he said.

The disparity in the projections by Aspin and Domenici arises largely from different assumptions about what size cut in long-term spending authority will be required to achieve specific spending reductions for a particular year.

But the message was the same from both: a decline for defense at the end of the 1980s that would be as precipitous as the increase at the start of the decade.

If the trend continues at the same rate through the five years leading to the Gramm-Rudman-Hollings target of a balanced budget, "we will end up with a 1990 defense budget below the levels of 1980 -- even after accounting for inflation," Aspin said.

"In other words," he added, "Gramm-Rudman could give us a defense budget that Jimmy Carter would assail as a threat to our national security."

The budget-balancing legislation, named for its chief sponsors, Sens. Phil Gramm (R-Tex.), Warren B. Rudman (R-N.H.) and Ernest F. Hollings (D-S.C.), sets mandatory targets for spending reductions that would cut deficits by roughly equal installments from $200 billion to zero by fiscal 1991.

If Congress does not achieve the goals, the president would be required to cut spending under a formula established in the law that splits the cuts equally between defense and domestic outlays.

Reagan endorsed and signed the legislation despite complaints from Pentagon officials that it would have a disastrous effect on the military buildup Reagan launched five years ago. Reagan has said he will continue to push for a 3 percent after-inflation increase, along with a recouping of losses from earlier congressional cutbacks.

But the law does not give him flexibility to exempt defense spending from cuts if the mandatory budget-cutting provisions take effect.

The prescribed cut for the rest of fiscal 1986, which ends Sept. 30, is relatively small: $11.7 billion, of which defense would account for half, or $5.85 billion.

To achieve the savings, however, new spending authority would have to be cut by a much larger amount because only a portion of what is appropriated is spent immediately, especially in the case of multi-year weapons procurement programs.

Thus Aspin estimates a spending authority cut of $10.3 billion to $13.3 billion for the rest of the current year. This would come on top of a near-freeze of defense spending approved by Congress for this year that virtually stalled Reagan's defense buildup. Under Gramm-Rudman-Hollings, spending for defense, even without accounting for inflation, will decline for the first time since 1973, Aspin said.

The projected cutback for fiscal 1987 would be several times larger to achieve a deficit target of $144 billion, as prescribed for the year by Gramm-Rudman-Hollings.

Domenici calculates that it will take a combined outlay cut of $60 billion to reach the target, including $30 billion from defense. Figuring that a cut of $3 in long-term spending authority is necessary to achieve $1 in immediate savings from defense, Domenici calculates that defense spending authority will have to be reduced by $90 billion.

Using a different ratio, Aspin comes up with a cutback figure of $65 billion for fiscal 1987 but acknowledges that it could be higher, especially if cuts are made largely in long-term weaponry contracts that offer little immediate savings.

Aspin's and Domenici's conclusions about the extent of the impending cutbacks are challenged by Gordon Adams, director of the defense budget project at the Center on Budget Policy and Priorities, an independent group that has been critical of the Reagan military program.

Adams contends that defense outlays will continue to increase despite Gramm-Rudman-Hollings because of a backlog of spending authority from previous years.