The House voted to boost this year's budget deficit by $13 billion to $42.8 billion yesterday in what Republicans characterized as a precursor of failure for Democratic budget-balancing efforts for 1981.

This curtain-raiser to congressional action on President Carter's proposed balanced budget for next year came as Carter on the one hand intensified his lobbying against more military spending increases in the House and, on the other, inadvertently strengthened the hand of the big defense spenders in the Senate.

House Speaker Thomas P. O'Neill Jr. (D-Mass.) said Carter told congressional leaders the Pentagon could not even spend all the $5.1 billion in extra defense money that some House members are proposing for next year.

But Carter's nomination of Sen. Edmund S. Muskie (D-Maine), the current chairman of the Senate Budget Committee, as secretary of state puts Sen. Ernest F. Hollings (D-S.C.) in line to assume the committee's chairmanship. Hollings was the architect of a big defense spending increase -- bigger even than House defense advocates are now proposing -- that the Senate Budget Committee approved earlier this month.

"I don't see any radical change" in the committee's outlook, Hollings told reporters, although he acknowledged that his priorities differ from Muskie's and reasserted his support for increased defense spending.

Asked if he would be "in sync" with the administration on spending priorities, Hollings grinned and said, "they can get in sync, I'm confident." p

Budget Committee aides said Muskie will probably manage the budget on the Senate floor but relinquish the leadership role to Hollings when the budget resolution goes to a House-Senate conference, where the real guns-versus-butter faceoff may occur if the two houses differ sharply over defense spending.

The House vote to increase the spending lid for 1980 was required because Congress breached its previously adopted ceiling last month, in effect freezing all spending bills until the ceiling could be raised.

Caught in the freeze were a number of major programs, including food stamps, which are scheduled to be cut off in June if new money is not allocated in the meantime.

On a generally party-line vote of 244 to 173, the House sanctioned a spending increase of $24 billion over the $547.6 billion that had been approved by Congress late last year as the budget ceiling for 1980.

This, in effect, raises the projected deficit from $29.8 billion to $42.8 billion.

House Budget Committee Chairman Robert N. Giaimo (D-Conn.) said the spending and deficit increases were dictated almost entirely by inflation and higher than expected interest rates. Most of the additional money, he added, is required by law.

The Budget Committee last month recommended a spending ceiling increase of $19.4 billion, but pressure for higher outlays was mounting so fast that Giaimo was forced to ask for $4.6 billion more. It was on this supplementary request that Republicans staged their protest, contending that any uncontrollable increases should be offset by spending cuts in other areas.

To do otherwise, asserted Rep. Delbert L. Latta (R-Ohio), ranking minority members of the Budget Committee, is to offer "proof that Congress is incapable of containing spending." Approval of a bigger deficit for 1980 raises serious questions about whether the 1981 budget can be held in balance in light of the probabilty of similar spending pressures next year, he added.

As the House began debate on 10 amendments to the proposed balanced budget for 1981, it voted down a proposal by Rep. James H. Quillen (R-Tenn.) to restore $1.7 billion in state revenue-sharing money by cutting foreign aid, science research funds and other programs. However, another amendment to restore revenue-sharing funds by offsetting cuts in categorical grants to the states is still pending.