The Senate Budget Committee last night broke its budget impasse and approved the broad outlines of a plan to reduce next year's budget deficit by $55 billion through freezing Social Security benefits, allowing defense spending to grow only with inflation and imposing some but not all of the domestic spending cuts proposed by President Reagan.

The vote was 11 to 9, with all Republicans voting for the plan and all Democrats against it, except for Sens. Charles E. Grassley (R-Iowa) and J. James Exon (D-Neb.), who voted present.

The plan rejects Reagan's proposal to cut federal workers' pay by 5 percent and instead would freeze both military and civilian pay next year.

It includes no tax increases.

The agreement, hammered out in a caucus of committee Republicans over the last day and a half, would produce a deficit of $172.3 billion next year, declining to $101.8 billion by fiscal 1988. The cumulative total of deficit reductions over the three years would be $296.7 billion.

This would virtually meet the target of the White House and Senate Republican leaders, which was to cut deficits by at least half to no more than $100 billion at the end of three years.

The Republicans' plan would cut next year's deficit by $7.2 billion more than Reagan's budget would have done, with substantially bigger reductions in future years.

In a testy debate after the plan was unveiled, Democrats angrily complained that the Republicans' claim of $55 billion in deficit reductions for next year, including about $20 billion from defense, was a "fraud" because the reductions were calculated from Reagan's military spending plans rather than congressional spending levels.

For next year, Reagan had requested a Pentagon spending increase of nearly 6 percent above inflation. The agreement would provide growth only to keep pace with inflation next year, estimated at about 4 percent, with increases of 3 percent above inflation allowed for the two following fiscal years.

Budget Committee Chairman Pete V. Domenici (R-N.M.) denied Democratic allegations that the Republicans were "jimmying" the numbers on defense savings and therefore total deficit reductions, saying "you're just dead wrong. We should all understand there's no fraud."

The major change in what was approved last night was inclusion of the freeze on Social Security benefits, under which cost-of-living increases for retirees would be eliminated for one year. Earlier, the committee deadlocked on the issue when it voted both to grant the inflation adjustment and to deny it. Also included under the cost-of-living freeze were other government retirement programs, including civil service and military pensions.

The Republicans' deficit-reduction plan was worked out in behind-the-scenes negotiations led by Domenici and was nailed down last night after the panel formally rejected Reagan's budget proposal and three alternatives that would have made deeper deficit reductions than would Reagan's draft.

Omission of tax increases, which some committee members had described as critical to breaking the budget impasse, came after Reagan earlier in the day intensified his opposition to tax increases.

"I have my veto pen drawn and ready for any tax increase that Congress might even think of sending up," Reagan told a conference of business leaders at the White House.

"And I have only one thing to say to the tax increasers: 'Go ahead, make my day,' " the president added, borrowing a line from a movie in which Clint Eastwood held a gun on a criminal and dared him to move.

There was some angry grumbling from Democrats, including Sen. Ernest F. Hollings (S.C.), who described the president's comment as "outrageous nonsense."

But, in votes during the day on the three budget plans offered as alternatives to Reagan's, including a Hollings proposal that had some tax increases, none got more than four votes from the committee's 22 members. Other tax-increase proposals were defeated by similarly lopsided margins on Tuesday.

The Republican plan adds deficit reductions of $22.5 billion to the total approved earlier by the committee as it went through the budget line by line, approving a freeze on most domestic spending but rejecting nearly all of Reagan's proposals for deep cuts in many programs and elimination of others.

In negotiations yesterday, Republicans on the committee stuck with a freeze but also accepted Reagan's proposal for termination of such programs as the Economic Development Administration, the Appalachian Regional Commission and the rural housing program.

While rejecting Reagan's proposal to end the $4.6 billion program of revenue-sharing with local governments next year, the committee agreed to phase it out, spreading next year's money over two years.

It voted to cut mass-transit capital and operating funds by 25 percent, cut the small business loan program by two-thirds, reduce postal subsidies, trim Amtrak funding from $700 million to $400 million over three years, reduce Urban Development Action Grants by 20 percent and slice $200 million out of the student loan program. Most of these programs have been targeted for elimination or considerably more drastic cuts by Reagan.

The committee agreed to freeze spending for farm programs rather than cut them as severely as Reagan had proposed.

Rather than eliminating the direct loan program of the Export-Import Bank, as Reagan proposed, the committee agreed to cut it from $3.8 billion to $1.2 billion.

The agreement also would freeze fees charged by doctors and hospitals under Medicare and provide for a gradual increase in premiums paid by patients.

It would freeze, rather than cut, Aid to Families With Dependent Children, the basic welfare program, but would allow for cost-of-living increases for Supplemental Security Income recipients.

Rejection of Reagan's budget came as no surprise, since the committee had voted by large bipartisan majorities to reject his proposed domestic spending cuts and military buildup plans for fiscal 1986.

Eight of 12 Republicans on the committee joined Democrats in voting against Reagan's budget, including Domenici, who had been leading the unsuccessful fight for most of the president's controversial domestic spending cuts. Only Sens. Rudy Boschwitz (R-Minn.), Orrin G. Hatch (R-Utah), Dan Quayle (R-Ind.) and Steve Symms (R-Idaho) voted for the president's budget.

In pushing for formal rejection of Reagan's budget, Exon said he was not trying to embarrass the president but rather was trying to avoid "total impasse" by prodding Reagan to help negotiate a compromise.

Domenici contended that slapping the president by rejecting his budget was unlikely to put him in a mood to compromise, and, after Reagan's comments on taxes, several Democrats said they feared that little more could be accomplished in the way of deficit reduction this year.

In Reagan's speech to the business people, he took the committee to task for failing to approve his domestic spending cutback proposals and defended his proposed increase in defense spending.

The key elements of the agreement reached last night by the Senate Budget Committee represent targets for inclusion in a congressional budget resolution, but the panel is expected to put teeth in the provision today by adopting "reconciliation instructions" ordering congresssional committees to make the spending cuts necessary to meet the targets.

The House Budget Committee is expected to begin work soon on a resolution of its own.