House Democrats reversed course last night and pulled back a popular proposal to restore funds for Medicare that could have nailed down a budget victory for President Reagan in the House.

After meeting with Democratic leaders, Rep. Henry A. Waxman (D-Calif.) said he was withdrawing an amendment to fatten a Reagan-backed Republican budget plan by $16.3 billion in Medicare spending over the next three years because he did not want the amendment to become "the engine that drives the [Republican] budget."

But he told reporters he had been assured by Democratic leaders that, no matter what budget finally passes, they will not allow committee approval of Medicare cuts beyond the $7 billion over three years that his proposal would have allowed.

Before Waxman decided to drop his proposal, many members of both parties said its passage as part of the Republican plan would vastly improve, if not assure, chances that the GOP budget would prevail over the other two versions.

The reason was that the extra Medicare money might attract enough votes among wavering Republican moderates, who otherwise would be inclined to support an alternative advanced by a group of bipartisan moderates.

Now the outcome is less clear.

Pending when the House recessed shortly before 1 a.m. was a proposal by Rep. Mary Rose Oakar (D-Ohio) to restore even more of the Medicare spending than Waxman would have done and finance the increase by a cut in defense spending. Oakar's plan was considered to have a lesser chance of passage than Waxman's did.

The Reagan-backed budget, the moderates' proposal and a third alternative proposed by the Democratic majority on the House Budget Committee are the rival budget plans facing the House as it nears a showdown, probably today or tomorrow, on a budget for the 1983 fiscal year, starting Oct. 1. The Senate passed its version of the budget last week.

Throughout the day, there was unease, spiced by some behind-the-scenes recriminations, in all camps over the Medicare proposal.

Some conservative Republicans balked at the idea of fattening their budget with Democratic add-backs, while some Democrats, including House Budget Committee Chairman James R. Jones (D-Okla.), were trying to convince Democratic sponsors of the Medicare proposal to withhold their amendment for fear it would put the Republican budget over the top.

All of the budget proposals are "in the same ballpark," Jones said, but "it's a big playing field," with more at stake than just funding for one program, he added.

Before the switch in strategy, Waxman said he was sympathetic to that argument but reluctant to agree to "a roll of the dice to see which [budget plan] wins and find that people on Medicare are the losers."

All of the three major budget proposals would cut Medicare, but the Republicans' budget, for which Reagan is lobbying by telephone from his California ranch, would cut by far the most -- $23.3 billion over three years, including $4.9 billion next year.

The other versions would cut $9.4 billion over three years, including $2.2 billion next year.

Waxman's proposal would have restored most, but not all, of the cut in the Republicans' budget, along with $1.7 billion for Medicaid and $1.4 billion for other health services over the same period. To offset these additional costs and avoid increasing the prospective 1983 deficit of more than $100 billion, Waxman would have increased taxes in the Republican party by $19.5 billion over the three-year period.

This would have brought the Republicans' tax increase proposal closer to those of the others, but it would still have been considerably lower -- nearly $115 billion over three years as opposed to $147 billion for the Budget Committee Democrats and $135 billion for the bipartisan moderates.

Earlier in the day, as the House waded through several inconclusive votes on defense spending, most of the main players agreed that the Reagan-backed budget appeared to have the edge in the final House budget showdown expected today or tomorrow -- but perhaps only if moderate Republican "Gypsy Moths" are appeased with restorations for programs such as Medicare.

Money for education, energy conservation, jobs and other programs was also being eyed by Republican moderates, but Medicare appeared to be the key test.

With the exception of Jones and some other Democrats, most members interviewed yesterday said the bipartisan moderates' alternative was more of a threat to the Republicans' budget than the committee's proposal.

The moderates' proposal is generally similar to the committee's version, except that it would cut taxes and defense a little less and domestic spending a little more.

The Medicare vote could "change the whole complex of what's happening out there," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), tacitly acknowledging concern about current prospects for the vote.

In the only successful Democratic proposal to restore domestic spending, Rep. Paul Simon (D-Ill.) won bipartisan support for adding $737 million to the Republican budget and a smaller amount to the committee proposal for education.

On the defense votes, the House overwhelmingly rejected proposals that would have increased military spending in the Budget Committee's plan and subtracted from military spending in the Republicans' plan. In some cases, the amendments were offered by foes of the plans, in other cases by supporters hoping to make them more acceptable to a majority of the House.

In arguing against beefing up defense spending in the committee's plan, Rep. Thomas J. Downey (D-N.Y.) ridiculed the notion that $214 billion for defense next year, as the commitment recommended, was insufficient."Only in the Land of Oz and in the Congress is $214 billion . . . some kind of denuding of our national defense," he said.

But, arguing for an increase in the figure, Rep. Dan Daniel (D-Va.) contended that the full amount of Reagan's defense request is needed to improve troop readiness.

On a vote to increase the committee's defense budget by $7.6 billion over three years, a move supported by Jones in hopes of making the budget more acceptable to defense-spending advocates, the tally was 339 to 83 against any change. An effort to pare the Republicans' relatively larger defense proposal was defeated on a voice vote.

But the votes were so muddled by members' impatience with the complex amending process, which was being used for the budget and was adopted to keep everyone equally happy or unhappy, that the result was less than meaningful.

At one point, Rep. Neal Smith (D-Iowa) complained to his colleagues that they were "just amending someone else's budget that you won't vote for anyway." The process was useful, he added, only to the extent that members can go home and tell constituents "we did something we didn't do."