The House yesterday decisively rejected a proposal to divert $5.1 billion from domestic to defense programs in next year's budget.

In a major victory for the Carter administration and a defeat for advocates of more money for the Pentagon, the House voted, 164 to 246, to stick with its Budget Committee's proposal for $147.9 billion for defense in an overall balanced budget of $611.8 billion for fiscal 1981.

The breadth of the margin came as a surprise to both sides. Especially after the ill-fated Iranian hostage rescue mission, a much closer vote had been expected, prompting last-minute lobbying by the administration, Budget Committee leaders, the House Democratic leadership and labor and other groups.

It was the first major guns-versus-butter floor vote in either house of Congress since the Iranian hostage seizure and the Soviet invasion of Afghanistan, although clearly not the final word on the subject.

The Soviet Budget Committee has approved military outlays of $155.7 billion for next year, largely at the expense of deep cuts in social programs, and the Senate is expected to go along when it takes up its budget resolution next week.

Differences will then have to be ironed out in House-Senate conference, where the succession of defense-minded Sen. Ernest F. Hollings, (D-S.C.) as Senate Budget Committee chairman, replacing Secretary of State-designate Edmund S. Muskie, is expected to strengthen the hand of military spending advocates.

House Budget Committee Chairman Robert N. Giaimo (D-Conn.) said he expects the House resolution to withstand other proposed changes and win final approval next week. He predicted that the House-Senate conference would produce a final defense figure of "somewhere above $150 billion," which is just about on the $150.5 billion target that Carter set in his March budget proposal.

The House proposal for defense represents an increase of about $14 billion over current spending. But Rep. Marjorie S. Holt (R-Md.) argued that it wasn't enough in light of recent military and foreign policy reversals.

Evoking memories of the aborted hostage rescue raid a week ago, Holt said the mission failed because of equipment failures and asserted, "If we are to undertake these missions, we must support them."

She also accused President Carter and Defense Secretary Harold Brown of opposing her proposal, which had been cosponsored by 155 Republicans and Democrats, "strictly for political reasons."

Both Carter and Brown had written letters to congressional leaders opposing Holt's amendment, and Carter invited about 100 potentially swing Democratic votes to the White House Monday night for some personal lobbying on this and other issues.

In response to Holt's arguments, Giaimo contended that the Budget Committee's spending blueprint was a painstaking effort to achieve a proper balance between domestic and military needs -- an assertion that a number of other lawmakers echoed in attempting to explain why the committee's proposal has withstood all proposed changes thus far.

"A lot of things are wrong with our military and our military posture," Giaimo argued. "But money isn't the most serious problem." He said the "will of the American people to be strong again" is the dominant concern and "no extra billions of dollars will make that happen." He also attempted to reassure wavering lawmakers that the defense figure could be increased later in the year if additional needs arise.

Another factor in the House vote was the domestic programs that Holt, joined by Rep. Phil Gramm (D-Tex.), proposed to cut in order to provide the extra $5.1 billion for defense while staying within the balanced-budget constraints that both Carter and Congress are demanding to help fight inflation. They cut a broad swath through the concerns of many of Washington's most powerful liberal interest groups.

The Holt-Gramm amendment would have cut public service jobs by $1.7 billion, reduced a variety of bureaucratic costs by $1.7 billion, slashed food stamps by $600 million, eliminated $600 million in foreign aid and saved $500 million by imposing a 20 percent across-the-board cut in the budgets of 17 regulatory agencies. In many cases, these cuts would have come on top of big reductions proposed by the Budget Committee.

Yesterday's vote did not, however, indicate a groundswell of sentiment for more domestic spending. On Wednesday, the House beat back four attempts by Democratic liberals to soften some of the Budget Committee's proposed domestic cuts, although it defeated the most modest of the four by a much smaller margin than the Holt-Gramm amendment got.

Giaimo, and others as well, interpreted the two days' vote as an indication that the committee had struck the proper balance for the House as a whole.

Rep. Leon Panetta (D-Calif.), another Budget Committee member, said he thought the vote on the defense spending increase would have been much closer, however, had it been held shortly after the failure of the hostage mission. He said time and greater knowledge of why the mission failed contributed to a calm response.

On the vote, 34 Republicans joined 212 Democrats in voting against the Holt-Gramm amendment, while 49 Democrats and 115 Republicans supported it. Among Washington-area lawmakers, only Holt voted for her amendment.