The House began beating back Republican deficit-reduction alternatives yesterday as leaders of both parties predicted passage today of a Democratic plan that protects cost-of-living adjustments for Social Security while freezing defense spending and restraining the growth of most domestic programs.

The Democratic plan, which would cut next year's projected deficit by $56 billion to $173 billion, was denounced by Republicans as filled with "smokescreens, magic and sleight of hand." But the GOP appeared to hold little if any hope of stopping it in the Democratic-controlled chamber.

In voting on the first GOP-sponsored alternative, the House rejected, 382 to 39, a proposal from conservative Rep. William E. Dannemeyer (R-Calif.) that would have gone beyond President Reagan's original budget in domestic cost-cutting.

Then the House voted by a slightly smaller margin, 335 to 87, against a plan offered by Republican moderates that would have included the defense freeze, but with different cuts in domestic spending, such as a freeze on all government pensions except Social Security.

Before recessing for the evening, the House also rejected, 361 to 54, a Congressional Black Caucus proposal that would have made additional deep cuts in defense spending, protected and in some cases expanded social welfare spending and raised taxes for the wealthy while cutting them for the poor.

In response to Republican protests that the caucus proposal would gut the Pentagon budget, Rep. Ronald V. Dellums (D-Calif.) contended Congress should have the "audacity" to curb weapons expansion in the interests of both peace and deficit reduction.

Rep. William H. Gray III (D-Pa.), chairman of the House Budget Committee as well as a member of the black caucus, voted present on the caucus proposal.

Pending for action today is a Republican leadership plan that would allow defense spending to grow with inflation while cutting deeper into domestic spending than the Democrats' plan, along with a proposal from Democratic moderates to increase the deficit-reduction total by raising taxes and freezing Social Security benefits.

The Democrats' plan parallels a budget adopted by the Senate this month in proposing to reduce deficits by $56 billion next year, but the Senate's spending cuts for future years are larger and the priorities in the two plans differ markedly.

The House Democrats would freeze defense spending and continue to allow Social Security benefits to grow with inflation, while the GOP-dominated Senate would freeze Social Security benefits and increase Pentagon spending to cover inflation. House Democrats propose smaller cuts in domestic spending and would eliminate only revenue sharing, while the Senate would terminate a dozen programs.

House Democrats apparently cleared the way Tuesday for passage of their plan with agreement on a compromise that would endorse a minimum tax on corporations without specifying whether revenues would be used to cut taxes or reduce deficits.

"I'd be greatly surprised if the budget wasn't adopted," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) as the two-day budget debate began. "We know where the votes are, we know how the votes are going to be cast," added Rep. Delbert L. Latta (R-Ohio), ranking Republican on the Budget Committee.

But Republicans nonetheless pounded on the Democrat plan, contending it was weighted too heavily toward retrenchment in defense spending and questioning the validity of some of its claimed savings from domestic spending.

A key theme of the debate, however, was that defense spending had to be restrained after four years of rapid growth.

Latta, in an otherwise staunch defense of increased spending for the Pentagon, acknowledged that recent reports of spending excesses have undercut support for Reagan's military buildup. "I'm among the first to acknowledge that the high-priced toilet seats, the high-priced wrenches have taken their toll," said the senior budget panel Republican.

Democrats were even more outspoken. "We have force-fed the Department of Defense over the last few years," said Rep. Thomas J. Downey (D-N.Y.).

There was an undercurrent of skepticism, especially among Republicans, about the deficit-reduction claims of both the Senate and House plans.

In testimony before the Rules Committee Tuesday, Rep. Buddy MacKay (D-Fla.) also questioned whether substantial deficit-reduction will occur. Citing signs of economic slowdown and increased inflation, MacKay said that even if the House Democrats' plan is approved "we may go home and find deficits are higher than they were when we started because the problem is running away from us."