The House resisted temptation yesterday and voted 242 to 182 against a quickie balanced budget as Republican leaders, joined by some Democrats, claimed that support is mounting on the House floor for the budget alternative that President Reagan prefers.

But they said Reagan may have to swallow more money than he wants for Medicare and perhaps some other social programs as the price for defeating budget alternatives drafted by Democrats on the House Budget Committee and a bipartisan group of moderates.

Both would give Reagan more than he wants in domestic spending and tax increases and less for defense, although the deficits from all three proposals would be in the $100 billion range for next year.

"We're on the margin. . . . Our toughest problem is how to cope with the Medicare issue," said House Minority Leader Robert H. Michel (R-Ill.) in asserting that his own budget proposal, which Reagan has endorsed, appears now to have a 50-50 chance of passage later this week.

Michel's budget is gathering strength, and amendments to restore funds for Medicare and education would "improve its chances that much more," said Rep. Leon E. Panetta (D-Calif.), a leading supporter of the bipartisan moderates' proposal.

"Right now the old coalition seems pretty much alive," added Panetta, referring to the coalition of Republicans and conservative Democrats that gave Reagan his big tax and budget victories in the Democratic House last year.

It was with anguish and mixed feelings that many House members passed up the chance to vote for a balanced budget, instead of an almost certain $100 billion deficit, for the 1983 fiscal year that starts Oct. 1.

Although a vote for the balanced budget amendment offered by Rep. John H. Rousselot (R-Calif.) could have been reversed later when the House got down to voting on the three main choices, it meant going on record for enormous domestic spending cuts, more than $450 billion over the next three years; these were how Rousselot arrived at a balance.

But some Republican leaders, nervous that passage of the proposal might jeopardize their own plan, contended that it was a futile, perhaps damaging gesture, and were visibly relieved when it failed. "It's sort of like the Zuni rain dance," said Rep. Barber B. Conable (R-N.Y.). "It makes the tribe feel better, but it doesn't bring rain."

On the House floor, however, the fight had all the earmarks of a holy crusade.

"Mr. Chairman, this is it," proclaimed Rousselot as he introduced the proposal, amid shouts of "Vote, vote, vote" from his colleagues. "We have talked about it, discussed it, rehashed it. Now we can vote for it."

Others may claim that their proposals will lead to a balanced budget sometime in the future, added Rousselot, but "the American people are asking why not now."

The reason, responded Rep. David R. Obey (D-Wis.), is that Rousselot used "phony numbers," including $38 billion in an assumed windfall from economic growth higher even than the administration projects, to help arrive at balance. What Rousselot does is "lay his hands on the deficit, like an old country preacher saying, 'Heal, heal, heal,' " said Obey.

Its proposed savings would never be achieved, added Panetta, and, if they were, "it would create chaos in society."

Responding to suggestions that they were asking for political trouble, proponents of the Rousselot plan said they were willing to do so. "That's what this country needs--some kamikaze pilots here in Washington," said Rep. Larry J. Hopkins (R-Ky.).

"We have promised and promised and promised a balanced budget at the end of the rainbow," said Rep. Daniel E. Lungren (R-Calif.), "but we spend and spend and spend and never get there."

For Rep. Denny Smith (R-Ore.), who defeated former Ways and Means Committee chairman Al Ullman (D) in 1980, it was a matter of mandates. "This is why I was elected to Congress," he stated in declaring his support for the Rousselot proposal.

But over-promising is a problem too, noted Panetta. "We promise magical answers and solutions and then we never deliver," he said.

On the Rousselot proposal, most Democrats opposed it, while most Republicans--although not including Michel and Conable--favored it. Forty-seven Democrats and 53 Republicans switched sides. All Virginia members voted for the Rousselot proposal, joined by three Marylanders, Reps. Marjorie S. Holt (R), Beverly B. Byron (D) and Roy Dyson (D).

As the House headed last night into a series of votes on up to 70 proposed amendments to the three major plans, most attention was on several proposals to restore funds for Medicare, which would be cut by $23.3 billion over three years under the Reagan-blessed plan.

The other two proposals would cut Medicare far less severely, and a major Medicare add-back was viewed as crucial to winning support of moderate Republican "Gypsy Moths" who are leaning toward the moderates' alternative.

But Michel, caught in something of a box, said he may have to fight the Medicare add-back proposals to keep from losing support of conservatives who oppose revenue increases that have been proposed to offset the spending increases.

The strength of the Michel proposal was underscored, however, by jostling within Democratic ranks.

House Budget Committee Chairman James R. Jones (D-Okla.) asked leaders behind the bipartisan moderates' proposal to withdraw their plan on grounds that it would only help win passage for the Reagan-backed plan. But they refused, and the House Democratic leadership declined to get involved.

Panetta said backers of the moderates' plan believe that theirs is the only viable alternative to Michel's proposal. Especially if efforts to restore Medicare funding fail, "it becomes a very different ball game because we draw many more moderates to our substitute" than the committee proposal does, Panetta contended.

Before recessing for the night, the House rejected, 237 to 175, an effort by Jones to raise the tax increases contained in Michel's budget for next year by $7.5 billion to finance extra money for education, employment, health, income security programs and social services. The vote was viewed as a setback to Jones' hopes for passage of the Budget Committee's alternative.

Other amendments were rejected, including a Republican proposal to reduce the Budget Committee's proposal for tax increases by about $10 billion next year to bring it in line with Michel's budget.