A tentative agreement on a defense-loaded balanced budget for 1981 collapsed in a House-Senate conference last night, threatening a serious disruption of the 6-year-old congressional budget process.

The understanding, worked out by the House and Senate Budget Committee chairman to break an earlier impasse, fell apart when five liberal House Democrats objected to the size of the chairman's proposed defense spending increase.

The agreement would have raised military outlays from the House proposal of $147.9 billion to $154 billion, $1.7 billion short of the Senate's original proposal of $155.7 billion.

The proposal would have made corresponding cuts in domestic programs, ranging from mass transit to public service jobs, and would have wiped out the House's proposed $1 billion surplus.

"They were nickel-and-diming us on social programs and giving away the store on defense," complained Rep. William M. Brodhead (D-Mich.), who was joined by Reps. Leon E. Panetta (D-Calif.), Norman Y. Mineta (D-Calif.), Richard A. Gephardt (D-Mo.) and Timothy E. Wirth (D-Colo.) in blocking the agreement.

The conferees agreed to return today for another attempt to reach a compromise, but House Budget Committee Chairman Robert N. Giaimo (D-Conn.) assessed the situation pessimistically. "We're very near a serious disagreement. If not at a serious disagreement," he told the conferees.

Senate Budget Committee Chairman Ernest F. Hollings (D-S.C.) laid the blame for the unraveling on House liberals, who, he said "reneged" on the package. Other sources said Panetta and Wirth first indicated they would go along with the package with misgivings but then, after a dinner break, re-emphasized their objections.

Without the five, Giaimo would have a bare majority of the House Democratic conferees behind him. But the split could well have presaged a bitter battle -- and perhaps defeat -- when the budget resolution came to the House floor for final approval.

The agreement closely resembled an amendment to increase defense spending by $5.1 billion that was offered on the House floor by Reps. Marjorie S. Holt (R-Md.) and Phil Gramm (D-Tex.) last month and resoundingly defeated.

Earlier yesterday, Giaimo indicated that the House conferees would be willing to go as high as $153 billion for defense if they did not have to make more concessions on critical social programs. But the Senate conferees indicated that $154 billion for defense was their absolute minimum, even though it was far from the kind of 50-50 split that conferees frequently make.

If no agreement is reached today, Giaimo indicated he would seek to break off the conference and take the issue back to both houses.

Sources indicated one course of action might be for the House to vote on its own final offer and then submit that version to the Senate, thereby testing whether Hollings and his conferees represent the real priorities of the Senate.

However, the Senate rejected proposals to increase social spending by trimming back defense increases by an even larger margin than the House rejected higher defense spending proposals.

Conferees have had to go back to their respective houses for instructions on budget resolutions before, but not on such a deep split involving the heart of the budget. Such an impasse could compound the economic and political strains threatening to foil President Carter's and Congress' joint drive for the country's first balanced budget in 12 years.

The guns-versus-butter impasse began to take shape toward the end of a marathon 13-hour session Monday, when the Senate conferees signaled that they would not budge far from their original defense proposal.

They came down by $300 million, while the House group indicated willingless to add $4.1 billion to the House defense proposal.

On domestic programs, according to Giaimo, the House came down $6.5 billion, while the Senate agreed to come up only $2.8 billion.