House-Senate conferees broke their budget deadlock yesterday and agreed on a nominally balanced $613.3 billion spending plan for 1981, but House Speaker Thomas P. O'Neill Jr. (D-Mass.) warned that the budget will almost certainly end up in the red.

The new compromise, which retains the Senate's heavy emphasis on defense while rejuggling figures slightly to satisfy political sensitivities on both sides of the Capitol, may be voted on by both houses today.

House and Senate, budget leaders were caustiously optimistic about the prospects, although House rejection of a previous compromise by a 242-to-141 vote this month clouded the outlook there.

Prompt approval would clear the way for action next week on $15 billion worth of long-delayed emergency spending bills for unemployment aid, disaster relief and other programs that have been running out of money because of a spending freeze caused by the budget impasse.

But relief about the breakthrough was tempered by public acknowledgement that the spending plan -- touted only three months ago by President Carter and congressional leaders as the first deficit-free budget in 12 years -- is being shoved out of balance as rising unemployment siphons off revenues while adding to social welfare costs.

Saying publicly what Carter administration officials are conceding privately, O'Neill told reporters that unemployment costs are rising so dramatically that "I don't see how we can have a second resolution that's balanced."

By this, O'Neill meant that Congress will probably be confronted by a deficit by late summer when it adopts a second, and presumably final, budget for next year. The first budget resolution currently before the two houses sets spending targets subject to revision to accommodate change in the economy.

O'Neill's prediction was in line with projections by some congressional budget experts of a $30-to-$40-billion deficit of unemployment, now 7.8 percent of the labor force, reaches 9 percent.

It also dovetails with an assessment by a well-placed administration source who said a $15-to-$20-billion deficit is possible if the unemployment rate is in the 8 1/4-to-8 1/2 percent range when the administration makes its midyear budget review next month.

However, Senate Budget Committee Chairman Ernest F. Hollings (D-S.C.) told reporters he thought a balanced budget is still possible and worth fighting for. "At the moment," the economic figures "don't look good . . . but I haven't given up at all on the pledge and struggle for a balanced budget," he said.

If the budget looks as though it's going to fall out of balance as the summer wears on, Holling said he would push for "across-the-board cuts . . . including defense" to keep it in balance.

It was Hollings, who has been unrelenting in his insistence on a big increase in defense spending, who finally broke the impasse yesterday morning by agreeing to an $800 million reduction in long-term spending commitments for defense.

The compromise retains $153.7 billion in outlays for defense next year, as the conference proposed in the earlier version of the budget rejected by the House. But they agreed to reduce budget authority for defense, which covers commitments that are made but not necessarily fully funded in a given year, from $171.3 billion to $170.5 billion. This was done largely to satisfy House Democratic liberals who fear that big long-term defense commitments will unduly squeeze social programs in future years.

The $800 million was shifted to categories covering a wide variety of social programs, possibly paying the way for more money for them in the future.

The conferees also agreed to reduce next year's already razor-thin projected surplus of $500 million to $200 million, using the extra $300 million for transportation and income security programs such as fuel assistance for low-income families, one of O'Neill's pet projects.

Hollings said he agreed to the defense concession only "to save the budget process," laughing off suggestions from the House that he bowed only after winning a big victory for renomination to the Senate in Tuesday's South Carolina Democratic primary.

Other sources said pressure to end the spending freeze was getting so strong that Congress was teetering on the brink of having to bend its budget rules to accommodate it. "Everyone felt the pressure and no one wanted to see the process break down," said Rep. William M. Brohead (D-Mich.).

The compromise won the support of O'Neill, the House Democratic Steering and Policy Committee and an influential group of moderate-to-liberal House Democrats who had opposed the previous compromise and were largely responsible for its defeat. The posture of Republicans and some Democratic liberals was in doubt.

President Carter had joined O'Neill in opposing the earlier compromise, protesting that social programs were cut to severely in order to accommodate higher defense spending. There was no immediate comment on the new agreement from the White House. House Budget Committee Chairman Robert N. Giaimo (D-Conn.) said Carter would be "ill-advised" if he didn't support the new version.

In addition to reaching agreement on the 1981 budget plan, the conferees voted to add another $400 million to the 1980 budget for disaster relief to cover the Mount St. Helens eruption, the Miami riots and other similar costs -- raising this year's deficit to $47 billion. A year ago Congress had projected a deficit of $23 billion for 1980.