A sharply divided Senate Budget Committee last night approved a $966.1 billion federal budget for next year after Republicans, over Democratic objections, won approval of stringent provisions to force the entire Congress to enact spending cuts aimed at cutting deficits by half over the next three years.

The budget was approved on a party-line vote of 11 to 10 after a quarrelsome debate in which Democrats accused the Republican majority of fudging figures to inflate deficit reductions and short-circuiting usual procedures to assure passage of unpopular spending cuts.

The committee's cuts include a Social Security benefit freeze and broad array of domestic spending reductions. The committee also stalled President Reagan's military buildup late Wednesday as part of a plan to cut $200 billion-plus deficits to $100 billion by fiscal 1988.

At issue yesterday was how to enforce the new spending limits.

Over Democratic objections that the enforcement provisions would ride roughshod over Senate procedures, the Republicans prevailed, 11 to 10, in resurrecting a "reconciliation" method of forcing legislative committees to cut programs to meet targets for spending reductions.

In addition to setting specific program-reduction limits for the committees, the budget panel voted to establish appropriations bills "caps" to keep spending for defense as well as domestic programs within targets set by the budget resolution for fiscal 1986.

Reconciliation procedures have been used before under Reagan to force spending cuts, but nearly always with controversial results, including complaints that they make it too easy to ram through program changes that would not pass on their own.

The spending caps would be new if imposed this year by both chambers. Last year the House balked at Senate-proposed spending caps, especially for domestic spending, but this year they are being touted by senators as a way to assure Defense Department spending restraint for the next three years.

The budget resolution now faces a difficult passage through the Senate. The Democratic-controlled House has yet to begin work on its budget, and after both chambers pass their plan it must be implemented through the reconciliation procedure and then the appropriations bills.

The budget would allow defense spending to grow next fiscal year only to cover the cost of inflation, a considerable reduction from Reagan's request for a 6 percent net increase after accounting for inflation.

It also would eliminate cost-of-living increases for Social Security and other retirement benefits next year and freeze most other domestic spending, including the pay of military and civilian employes of the government.

It incorporates some but not all of the program cuts and eliminations that Reagan recommended in his budget request. Programs such as the Economic Development Administration and Appalachian Regional Commission would be killed; revenue sharing would be phased out; programs ranging from Amtrak to Medicare would be cut sharply.

The committee's budget includes no tax increases, largely because of Reagan's opposition to such a move and Democratic skittishness over defying him on it.

Committee Republicans calculate that their budget would produce a deficit of $172.3 billion next fiscal year, declining to $101.8 billion after three years, virtually meeting Senate leaders' goal of deficits no higher than $100 billion by fiscal 1988.

But the committee also approved a proposal offered by Sens. Gary Hart (D-Colo.) and Howard M. Metzenbaum (D-Ohio) calling on the Senate Finance Committee to act on tax-simplification legislation as soon as possible. Hart said committee approval of the proposal sends a signal that "tax reform should not be held hostage to progress on deficit reduction."

But the partisan quarrel over whether Republicans had inflated their deficit-reduction claims continued yesterday. Using different economic assumptions than the Republicans did and calculating defense savings off congressional as opposed to administration benchmarks, the nonpartisan Congressional Budget Office said next year's deficit reductions amounted to $39.3 billion, compared with the GOP claim of $55 billion.

Sen. Lawton Chiles (Fla.), ranking Democrat on the committee, said the CBO figures amounted to "truth in budgeting," while Budget Committee Chairman Pete V. Domenici (R-N.M.) said the congressional figures did not take into account economic benefits, such as lower interest rates, that would result from reduced deficits.

The committee's work drew mixed reviews yesterday with most of the key players attempting to put some distance between themselves and the specific details of the budget plan.

Presidential spokesman Larry Speakes said the White House is "disappointed in some elements of the package," including the defense cutbacks and the fact that the committee did not cut more from domestic spending. But he said the administration was pleased that the budget included no tax increases.

Speakes made no mention of the Social Security freeze until asked about it. He noted that Reagan first opposed a freeze and then said he would consider one only if there was a "congressional mandate" for it. Speakes said the 11-to-9 party-line vote Wednesday night for the deficit reductions could not be considered such a mandate.

Senate Majority Leader Robert J. Dole (R-Kan.) endorsed the Social Security freeze but said the overall budget "needs some work," hinting at the possibility of three-way negotiations with the White House and congressional Democrats to assure its passage.

As a whole, he praised the committee's effort as a "good step forward."