The House Budget Committee voted 21 to 12 along party lines yesterday to approve a deficit-reduction plan for next year that would freeze defense spending and reduce many domestic programs but not tamper with Social Security.

Approval by the Democratic-controlled panel came after a move to reach bipartisan agreement on spending cuts fizzled in a dispute over defense funding levels and Democrats brushed aside Republican proposals to bring the plan more in line with one approved by the GOP-led Senate last week.

Both Democratic and Republican leaders indicated that the committee's plan -- which would cut deficits by $56 billion next year and $259 billion over three years -- would probably be approved when the full House votes on it next week.

The House Democrats' plan would cut spending for next year by the same amount as the Senate Republicans' version, although the Democrats would reduce deficits by less in future years and squeeze more from defense than from domestic programs.

In one key difference, the Democratic plan would freeze defense spending but allow Social Security benefits to grow with inflation, while the GOP plan would freeze Social Security but allow defense spending to rise with inflation.

President Reagan warned Congress last night that he would abandon the compromise he agreed to last week on defense spending if the House tries to cut it any further. Reagan told a party fund-raising dinner that he had "compromised greatly" by agreeing to the Senate plan.

The House Democrats' plan also would cut domestic agencies' spending less than the Senate Republicans' plan and kill only one of the dozen programs the Senate had proposed for extinction: revenue sharing with local governments. The Democrats' plan would freeze or cut back many of them, ranging from farm supports to Amtrak subsidies.

In a series of closed negotiations, rare for the sharply divided committee, Democrats and Republicans tried to compromise on defense and emerge with the committee's first bipartisan budget.

But the attempt foundered amid conflicting objections from Republicans and the House Democratic leadership. The committee's two-day flirtation with bipartisanship ended in a basically partisan vote. There were no Democratic defections and only one from GOP ranks: Rep. Henson W. Moore (R-La.) voted for the Democratic plan.

After the vote, Budget Committee Chairman William H. Gray III (D-Pa.) said the panel cut deficits by more than Reagan had proposed and, "I think we've done it with balance . . . , with fairness."

However, Senate Majority Leader Robert J. Dole (R-Kan.) denounced the House Democrats' plan as full of "smoke and mirrors" and said it fell at least $40 billion short of the three-year savings approved by the Senate. It made defense the "whipping boy," he said.

Both plans would freeze federal workers' pay, although the House rejected federal retirement cutbacks endorsed by the Senate, including a one-year pension freeze.

Low-income programs such as Medicaid, compensatory and handicapped education, student aid, Head Start, veterans health-care, child nutrition and job training would be allowed to grow only for inflation.

Both the House committee and the Senate plans freeze Medicare payments to doctors and hospitals. Increased costs for beneficiaries approved by the Senate were rejected by the House committee.

Both plans cut farm programs, although the House committee's reduction of $10 billion over three years is only two-thirds of the Senate's proposal for $15 billion in savings over the same period.

Among other programs that the House committee would cut, although by less than the Senate, were rural housing (20 percent), small business (20 percent) and Amtrak (10 percent).

Also slated for 10 percent cuts were a variety of community, regional and economic development programs, including the Appalachian Regional Commission, Economic Development Administration, the Community Development Block Grant program and Urban Development Action Grants (UDAG). UDAG was originally listed by the Senate as one of the programs it was terminating next year; Dole said yesterday that it would be cut 20 percent next year and killed in fiscal 1987.

Export-Import Bank direct loans, which the Senate would end, would be continued but limited by the House committee. The Strategic Petroleum Reserve would continue to be filled under the House panel plan; the Senate would impose a moratorium. Both plans would impose some new user fees. Neither proposes tax increases.

House Majority Leader James C. Wright Jr. (D-Tex.) and Rep. Delbert L. Latta (Ohio), ranking Republican on the budget panel, said the committee plan is likely to win House approval when it comes to the floor, probably Wednesday or Thursday.

"We live to fight another day," Latta said, meaning an eventual House-Senate conference on the budget is expected to resolve most critical issues.

However, he said the Republican leadership still plans to offer its own alternative. Moderate Republicans are also expected to offer an alternative. The Black Caucus will offer its plan, and conservative Democrats plan to offer their $75 billion deficit-cutting package that would freeze Social Security and incorporate a minimum tax.