The Senate yesterday endorsed a balanced budget for next year after sidestepping a Republican proposal for more radical spending surgery.

At the end of a day-long debate filled with political shadow boxing, senators -- on two basically party-line voes of 52 to 45 and then 56 to 41 -- fell in line behind President Carter's budget-balancing strategy for fighting inflation.

The votes had the effect of shelving a proposal advanced earlier by 46 Republicans and moderate-to-conservative Democrats to limit fiscal 1981 spending to 21 percent of gross national product and thereby restrict spending more heavily than Carter has proposed. They say the extra savings would permit tax cuts, while Carter is proposing tax increases.

But Senate Budget Committee Chairman Edmund S. Muskie (D-Maine) charged that a 21-percent-of-GNP limitation would force a cut of $45 billion from Carter's January budget, or $29 billion from his new balanced budget proposal -- requiring what Muskie called "radical reductions" in many politically popular programs.

Carter's budget amounts to about 22 percent of GNP, which is the total value of goods and services produced in the country during a year.

Although Sen. William V. Roth (R-Del.), chief sponsor of the 21 percent proposal, said Muskie's figures were exaggerated, many of Roth's original Democratic backers peeled off and voted for the balanced budget resolution.

This proposal was drafted by Muskie and the Senate Democratic leadership to prevent the political embarrassment of Roth's winning and claiming some of the current austerity limelight for the Republicans, as well as boxing congress into more extensive budget cuts than the White House or the Democratic congressional leadership wanted.

Although the resolution is not binding on future authorization and appropriations bills, it started the "sense of the Senate" that the Budget Committee should draft a balanced budget for 1981 and reserve any surplus for a tax reduction.

It also directed the Budget Committee prepare a list of further reductions that the Senate, if it wants, can adopt to bring spending within the 21 percent figure -- a list that is expected to be so draconian that many lawmakers will shy away from it.

As an additional sweetener in case of a surplus, the proposal was amended on the floor to specify that the money be used for business tax cuts to spur productivity and for offsetting next year's scheduled Social Security tax increases.

The resolution is in line with spending targets adopted last week by the House Budget Committee and is expected to prevail in the Senate Budget Commitee when it begins preparing its targets this week.

However, both Carter's revised budget and the $16.5 billion in cuts approved by the House committee drew fire yesterday from the Congressional Black Caucus of 15 House members, who vowed to submit a counter proposal of their own.

"We cannot and will not accept a budget which is indeed the budget of extreme conservatism," said Caucus Chairman Cardiss Collins (D-Ill.) in the latest expression of distaste among House liberals for budget cutting that focuses primarily on social programs.

During Senate debate on the proposed budget limits, Roth charged that Muskie's proposal was a "smoke screen to hide an effort to balance the budget with massive tax increases," which he said would total about $90 billion from all sources over the next fiscal year. It would, said Roth, amount to the biggest tax increase in history and "let the big spenders off the hook while throwing the country deeper into a recession.

Denying Roth's charges, Muskie said the Senate Budget Committee, under his resolution would cut $13 billion "and maybe more" from Carter's original budget, which he described as substantial.

"If senators don't believe that, wait until constituents [are heard from] . . . if you haven't felt that yet, then you're living in a smoke screen world," said Muskie.

He argued that Roth's proposal would mean finding $29 billion to cut from only about $47 billion worth of controllable outlays, including funds for revenue sharing with local governments, food stamps, aid to the poor, energy programs, the space shuttle, most aid to higher education and the foreign aid program. The only alternative to virtually dismantling these programs, he added, would be cuts in programs such as national defense.

Roth lost his chance to get an up-or-down vote on his proposal when the Senate refused, 52 to 45, to table Muskie's resolution. This set the stage for approval of the Muskie version by a vote of 56 to 41. On the critical tabling motion, the only Washington-area senator to vote with Muskie was Sen. Paul S. Sarbanes (D-Md.).

The debate had the aura of a political event from the start, with Roth and his supporters sporting lapel buttons reading "21 perent -- balanced budget and tax cuts too." By the end of the debate the rhetoric had escalated to the point that Roth was calling Muskie's proposal a "cup of poison" and Senate Majority Leader Robert C. Byrd (D-W.Va.) called Roth's proposal "snake-oil politics."

Meanwhile, in the House, two authorizing committees went the other way and voted a combined $1.9 billion increase in Carter's budget -- this time for farm loans and aid to Israel.

Under election-year pressure from farmers, the House Agriculture Committee approved a $1.7 billion boost in loans for farmers to offset losses from the Soviet grain embargo.

Separately, the House Foreign Affairs Committee approved $2.2 billion in U.S. aid for Israel -- about $200 million more than Carter requested.

And the House Armed Services Committee voted to authorize $3.3 billion more in military spending than the president has sought, mostly to buy planes, missiles, tanks and other weapons.

The panel is scheduled to act today on proposals that would authorize another $2.3 billion over the president's budget request, this time for ships and research. The votes are certain to intensify protests by liberals.