The long-smoldering congressional battle of the budget threatens to explode in the House today as liberals, disgruntled over new concessions to the Pentagon, ready to fight to defeat a compromise House-Senate budget resolution.

Although House leaders said they still expect the measure to pass, all sides conceded late yesterday the vote is likely to be close and, if Republicans join with the liberals, could end up in defeat of the compromise measure.

The challenge is being mounted by Rep. David R. Obey (D-Wis.), a member of the House Budget Committee, with the backing of the liberal Democratic Study Group. Obey contends the compromise measure reverses the House's priorities on defense.

The guns-vs.-butter dispute has been a focal point of budget debate since President Carter last January proposed increasing defense outlays sharply at the expense of some traditional social programs.

The House voted in its version of the measure to slash Carter's defense budget by $3.1 billion and boost spending for social programs, but the Senate approved his budget intact.The compromise leans toward the Senate's figures.

Yesterday, the Democratic Study Group sent out letters over Obey's signature urging House members to reject the compromise proposal on grounds that it "doesn't adequately reflect the actions of the House."

Interviews yesterday with a random sampling of congressmen likely to be swing votes showed many members still undecided about which way to go.

Rep. Robert N. Giaimo (D-Conn.), chairman of the House Budget Committee, and other key leaders spent most of the day yesterday trying to line up votes for approval of the resolution. If the measure is defeated, it will be redrafted.

A defeat on the House floor today would mark the first successful challenge to the Budget Committee's recommendations this year. Although the House debated extensively on the resolution, leaders fended off virtually all major attacks.

A key question today will be which way Republicans vote. GOP members traditionally have opposed the budget measure as a protest over spending, but they could back it this morning to preserve the higher defense outlays.

The compromise resolution calls for government outlays of $532 billion in fiscal 1980, with a deficit of $23 billion- $6 billion less than President Carter proposed in January.

For Fiscal 1980, the measure would provide $124.2 billion in defense spending, just $1.8 billion less than Carter requested. Authority to commit Pentagon funds for coming years would total $136.6 billion, $1.6 billion less than Carter asked.

At the same time, the compromise measure would trim some of the money the House had restored for education and other social programs, and would dilute a House mandate to eliminate revenue-sharing grants to states.

As is the custom in the House, members approved a series of largely symbolic amendments calling for across-the-board cuts in spending, repeal of oil company tax breaks and similar measures.

However, most of these were swallowed up during the negotiations among the House-Senate conferees. The budget process was designed to set limits on broad spending categories, not to deal with specific line items.

Defeat of the budget resolution today would not be the first time the measure has been rejected by the House. Budget votes in the chamber almost always are close and the House defeated the resolution in 1978.

The Democratic Study Croup's calculations show the compromise resolution has a $1.9 billion higher authority for defense spending than the House voted in its own version of the measure, and $1.8 billion less in new authority for social programs.

However, it's not entirely clear whether the conference committee could reach agreement on a new proposal very easily. The Senate stood firm on the defense figure during the previous conference, and is likey to do so again.

The two houses already are far behind schedule in working out a budge resolution. Under the new congressional budget process, the lawmakers were supposed to have set tentative spending targets by May 15.

However, the measure was delayed by continued wrangling and a spate of largely symbolic amendments in both houses-developments some analysts said reflect the conflict between the austerity mood and line-item projects.