House liberals won a stunning victory in the guns-vs-butter batter yesterday as the House overwhelmingly rejected a compromise fiscal 1980 budget resolution. The compromise would have allowed more for defense and less for social programs than the liberals wanted.

The vote was a lopsided 260 to 144 as 152 disgruntled Democrats abandoned the leadership and joined conservative Republicans in opposing the measure. Another 15 Democrats refused to vote at all.

It was President Carter's second major rebuke in the House in two days. The measure defeated yesterday essentially paralleled the president's budget recommendations. On Tuesday, the House Democratic Caucus rejected Carter's oil-decontrol plan.

The size of the opposition yesterday surprised even the liberals. Rep. David R. Obey (D-Wis), who led the challenge to the budget resolution, said later, "I didn't know it would be that big."

House leaders were at a loss to explain the size of the opposition vote. However, some members suggested the liberals had provided an opening for long pent-up frustration that had remained dormant during previous debate.

Rep. Barber Conable (R-N-Y.) la mented after the vote: "We seem to have developed a herd instinct here. We're stampeded by symbols, and that's what happened today."

A few hours after the House action, the Senate voted, 72 to 17, to amend its version of the budget resolution by adding $350 million for education and traning programs as a gesture to the House liberals. However, senators refused to cut the defense budget further, making the reaction in the House uncertain.

House conferees had demanded that the Senate add $300 million to the eucation and training budget and cut defense by $200 million. Both actions would involve long-range authority for commitment of funds and would not affect fiscal 1980 outlays.

The liberals' demands yesterday were largely symbolic. Budget officials said the money involved in both categories was so small that any changes now would likely be swallowed up by re-estimates later in the year.

However, failure to approve a budget resolution quickly could tie up congressional action on all money bills. Under present law, until Congress sets its initial spending targets, no house may pass an appropriations bill.

Yesterday's vote followed intense lobbying by organized labor and state educators fearful of reduced education grants. In fact, however, under the budget process, appropriations committees are free to allocate the money as they choose, in line with overall targets.

It was not immediately clear whether the Senate's counterproposal would attract enough votes in the House to pass the budget resolution. Tentative planare to bring it up again in the House sometime today.

Several House members expresed apprehension that yesterdayhs rejection of a compromise might make it virtually impossible for House leaders to win approval of any budget plan this year.

Said Rep. Richard Gephardt (D-Mo., a member of the Budget Committee: "You have to have a cut in defense, but that in turn causes you trouble on the other end of the spectrum."

Yesterday's vote was a thorny one for Republicans, who were caught between a general desire to approve higher outlays for defense and a historical opposition to high budget deficits. Yesterday's measure called for a $23 billion red ink figure.

All told, 108 Republicans voted against the resolution, 36 voted for it and 15 did not vote.

It was the first time in the five-year history of the budget process that the House has rejected a compromise budget resolution. Last year, the House defeated its own Budget Committee's recommendation but later accepted the version worked out by House and Senate conferees. CAPTION: Picture, REP. DAVID R. OBEY . . . even the ringleader is surprised