The Republican-controlled Senate Budget Committee yesterday rebelled against a continuation of President Reagan's military buildup and voted 18 to 4 to allow next year's Defense Department budget to grow only enough to keep pace with inflation.

In the first congressional vote of the year on deficit reductions, the committee approved spending targets that would slash Reagan's proposed military outlays of nearly $1 trillion over the next three years by $79.3 billion, starting with $11 billion in fiscal 1986.

The committee's action represented the heaviest blow to the Pentagon budget since Reagan began his defense buildup four years ago by winning congressional approval for an average after-inflation increase for defense of about 9 percent a year.

The president had requested a 6 percent after-inflation increase for the next fiscal year, refusing to compromise and saying as recently as Saturday in his radio address that "as long as I'm president we are not going back to the days when America was fast becoming an impotent democracy, too weak to meet its defense commitments."

Under the Budget Committee's plan, spending authority for defense would rise from $293 billion to $303 billion next year, but the increase would cover only the costs of inflation, estimated at about 4 percent for the year.

The plan would allow "real," or after-inflation, increases of 3 percent in each of the following two years, which also is less than Reagan requested although not as dramatic a cut as the committee approved for fiscal 1986.

The plan came from the Democratic side of the table under sponsorship of Sen. Ernest F. Hollings (D-S.C.).

But it was backed by two-thirds of the Republicans on the panel, including its chairman, Sen. Pete V. Domenici (R-N.M.), who had proposed a 3 percent after-inflation increase only the day before.

The only senators voting against it were Republicans Orrin G. Hatch (Utah), Robert W. Kasten Jr. (Wis.), Dan Quayle (Ind.) and Steve Symms (Idaho).

All Democrats on the committee supported the proposal after a move to freeze spending at this year's level without any increase for inflation failed 14 to 8.

Although the committee-approved figures represent only targets for inclusion in a nonbinding budget resolution, Domenici told the committee he would push for legislation that would "to the maximum extent possible" nail down the proposed limits for the next three years.

Yesterday's vote offered the first positive sign so far that Senate Republicans, who failed in two months of intensive efforts to reach an accord on deficit reductions, may yet be able to put together a package, although major difficulties remain.

"It's the beginning of the beginning," Senate Majority Leader Robert J. Dole (R-Kan.) said after the vote.

Domenici was expected to use the big defense cut as leverage to force major domestic spending cuts as well, including an across-the-board freeze on all programs but those targeted at the poor, augmented by more severe selective cuts proposed by Reagan in his budget.

The fact that eight Republicans joined all 10 Democrats in supporting the Hollings proposal also indicated that budget deals could be cut on a bipartisan basis, raising risks for the administration.

It may turn out to be significant that a majority of Republicans and Democrats pulled together on defense, despite an acrimonious fight earlier in the day over whether to use administration economic projections that Democrats contended would lead to an underestimate of future deficits.

Republicans prevailed on a party-line vote on that issue, and Democrats were infuriated. Domenici then fudged the issue by agreeing to use the administration's projections and less rosy ones prepared by the Congressional Budget Office and favored by the Democrats.

Even before the committee's action on defense, Dole suggested strongly that it was time for the White House to compromise on military spending.

Urging that the Budget Committee send its budget proposal to the floor "without recommendation," he said congressional leaders should then meet with Reagan and Defense Secretary Caspar W. Weinberger to work out a compromise.

Reagan and Weinberger steadfastly have refused to budge on their defense proposal, which was a major factor in failure of the Senate Republican leadership to nail down an overall deficit-reduction plan.

But until the Budget Committee vote the administration had little but rhetoric from Senate leaders to cause it to compromise.

Senate leaders can be expected to use the committee's action to renew their pressure for more flexibility from Reagan and Weinberger on defense.

The Budget Committee's vote yesterday stands in stark contrast to its posture last year, reflecting strong pressure from both sides of the aisle in Congress to force the Pentagon to bear its share of deficit reductions, even at the expense of a slowdown in Reagan's military buildup.

Last year senior committee members, including Domenici, stalled a congressional compromise on defense for several months as they held out for an after-inflation increase of more than 5 percent.

Because Hollings' proposal would cut about $20 billion from what defense spending would be without any congressional action, it gets the Budget Committee about one-third of the way toward its goal of about $60 billion in deficit reductions for next fiscal year as the first installment of a three-year plan to cut deficits by half to less than $100 billion.

The rest of the savings would come from domestic programs, including a possible freeze of Social Security benefits, as Domenici proposed in a comprehensive budget draft on Monday.

With a smaller cutback in defense spending than the committee approved yesterday, Domenici's plan would produce deficit reductions of $50 billion to $60 billion next fiscal year, which he contends would be enough to reach the three-year target.