The Democratic-controlled House yesterday approved a budget that matches the Republican Senate's proposed $56 billion in deficit reductions for next year but differs on defense and Social Security and could spell trouble in a House-Senate conference next month.

Overcoming divisions that had wracked their party, Democrats capitalized on GOP disarray to win an impressive 258-to-170 victory for a $967 billion fiscal 1986 budget resolution that freezes defense spending and restrains growth of domestic programs but does not tamper with Social Security.

Because the Senate voted to freeze Social Security benefits but allow Pentagon spending to grow with inflation, the two politically potent issues are expected to dominate negotiations on a compromise over the two chambers' separate plans.

House Budget Committee Chairman William H. Gray III (D-Pa.) said he interprets a series of House budget votes over the last two days as a bipartisan mandate to hold the line against Social Security cutbacks, but he did not rule out compromise on the issue.

Similarly, Senate Majority Leader Robert J. Dole (R-Kan.) has said he is "almost" prepared to see negotiations break down rather than give more ground on defense. But yesterday he played down that issue in a statement attacking the House plan as toothless in enforcing domestic cuts.

Gray and Dole said they are "hopeful" about a compromise. Pressure is strong in both chambers, and buttressed by a kind of partisan one-upsmanship, to make a breakthrough in curbing budget deficits that exceed $200 billion annually.

Although the House never voted separately on Social Security, it rejected, 372 to 56, a proposal from moderate Democrats that would have enlarged the deficit-reduction package by freezing Social Security benefits and raising taxes other than income levies by $54 billion over three years.

In an impassioned plea, Rep. Claude Pepper (D-Fla.), champion of the Social Security system, said the freeze would violate "solemn commitments" of both parties' leaders and "take bread out of the mouths" of the elderly. Freeze proponents argued in vain that it was the only way to trim burgeoning benefit-entitlement programs.

"Only a budget we all hate can do the job," said Rep. Marvin Leath (D-Tex.), principal sponsor of the proposal. "The president hates this budget. The speaker hates this budget. That means it's probably a pretty good budget."

The impact of the vote on Leath's proposal was muddled by inclusion of tax increases, rejected by President Reagan and Democratic leaders who generally feared political repercussions.

In a subsequent vote, the House again demonstrated skittishness about taxes by rejecting, 283 to 142, a watered-down leadership compromise to endorse a minimum tax on corporations without specifying whether revenues would be used to reduce deficits or to cut income tax rates.

Some members said that issue too was obscured by arguments in the debate and by a sense that compromise had rendered the provision meaningless.

Also defeated yesterday before adoption of the Democratic plan was a Republican leadership alternative that, while preserving cost-of-living adjustments for Social Security, would have cut more deeply into domestic spending and allowed defense spending to grow with inflation, as the Senate plan does.

The GOP leaders' plan was rejected, 329 to 102, with more than 40 percent of Republicans voting against it, including moderates, some conservatives and many farm-state legislators. Defeat of the leadership plan followed even more emphatic rejection Wednesday of two GOP alternatives, one backed by party conservatives and the other by moderates and liberals.

In a departure from Reagan's first term, when a relatively united GOP picked up enough conservative Democratic votes to control the House on budget issues, more Republicans than Democrats crossed party lines yesterday.

Twenty-four Republicans, mostly northeastern and midwestern moderates, joined 234 Democrats in voting for the Democratic plan, while 155 Republicans and 15 Democrats voted against it.

Nearly all of the "Boll Weevil" Democratic defectors of earlier years stayed with their party, reflecting efforts by the party leadership to include them in the process.

Washington area lawmakers voted along party lines. On the GOP leadership plan, only Virginia Republicans Stanford E. (Stan) Parris and Frank R. Wolf broke party ranks, voting against it.