A proposed Democratic budget that calls for higher taxes, more defense cuts and smaller domestic program cutbacks than President Reagan wants was approved yesterday by the House Budget Committee on a party-line vote that compounds its probable difficulties on the House floor next week.

"The only way I see it will pass is a joint effort on both sides of the aisle," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) before the 17-to-12 committee vote in which Republicans were united in opposition, conservative Democrat Phil Gramm (Tex.) abstained and liberal Democrat David R. Obey (Wis.), while voting for the fiscal 1983 budget resolution in committee, said he would lead a drive for an alternative on the House floor.

But O'Neill's gesture to House Republicans did not extend to Reagan, against whom he resumed his war of words. "He's just an unbelievable guy to be dealing with . . . ," said O'Neill. "He doesn't really know there's a recession out there. It's almost inconceivable he could be that isolated."

And O'Neill, who knows what it's like to lead a party in disarray, took solace from the Republicans' problems. "Talk about disarray," he said. "They're in complete disarray!"

House Budget Committee Chairman James R. Jones (D-Okla.) refused to concede defeat in advance of the committee's budget vote, but acknowledged that the lack of Republican support in committee hurt its chances on the floor. He vowed to "keep coming back relentlessly" until a budget is approved. But Republicans were having their troubles, too.

In the Senate, at least seven GOP moderates said they will seek major revisions in the budget resolution proposed by the Republican-controlled Senate Budget Committee and endorsed by Reagan. The changes include more defense cuts and tax increases as well as abandonment of the controversial proposal for $40 billion in Social Security savings over the next three years.

Additional funds for Medicare, Medicaid and college student loans were also proposed by the group, led by Sens. David Durenberger (R-Minn.) and John H. Chafee (R-R.I.).

If Democrats lose no more than a couple of votes against the Social Security changes, seven Republican defections would be enough to doom any cuts in benefits as part of the deficit-reduction drive, severely undercutting Reagan's latest budget initiative.

The Senate is scheduled to begin debate on the budget today, with the House following next Friday.

In the Democratic-controlled House, where the major budget battles were decided last year, all of them in Reagan's favor, the intervening week will be used for intensive bargaining--a "bidding war," some are already predicting--for support of conservative Democratic "Boll Weevils" and moderate Republican "Gypsy Moths."

House Republican leaders, who have spurned the Reagan-backed Senate plan, are putting together a budget of their own, working with some Democratic conservatives. The outcome is in doubt, and many lawmakers are girding for one or more votes against any budget on the table.

Even passage of a budget by both houses does not assure the kind of deficit reductions that Congress is seeking to help lower interest rates and prompt economic recovery. Both the proposed Senate and House budgets include "reconciliation" instructions to committees to raise taxes and cut spending in accord with the overall budget targets, although the House enforcement provisions are weaker, largely because of objections from its Appropriations Committee. Efforts to strengthen them on the floor are expected.

The proposed House budget anticipates spending of $780.5 billion for the 1983 fiscal year starting Oct. 1, with a projected deficit of $103.9 billion, or slightly less than the Senate's proposed deficit of $106.1 billion for a budget of roughly the same size. The deficits, calculated at $182 billion for this year if no new spending cuts or tax increases are imposed, would decline by 1985 to $34.6 billion under the House plan and $39.5 billion under the Senate plan.

The House committee's plan calls for $147 billion in tax increases over three years, $52 billion more than the Senate committee proposed. Both plans include a three-year freeze on domestic appropriations, but the House committee sweetened it with $8 billion in "add-backs" to popular programs, many of them aimed at relieving the effects of the recession. The House plan cuts benefit entitlement programs but not as severely as the Senate plan and leaves Social Security untouched. It cuts the huge defense build-up program by $47 billion through 1985, more than double what it would be cut under the Senate plan.

While the Senate plan freezes federal pay and pensions for next year, with 4 percent annual increases thereafter, the House plan provides for a 4 percent annual increase starting next October.

In all, the House proposal would make $418 billion in deficit reductions over three years, roughly similar to the $416 billion cumulative figure in the Senate proposal, although relatively more would come from tax increases, as opposed to spending cuts, under the House plan.

But it departed so far from the traditional Democratic budget approach that Obey and other liberals are threatening to oppose it on the floor, perhaps enough to kill it if conservative Democrats defect as well, as many are expected to do. Obey said yesterday that he objects to offering so many concessions so early, adding that it's like "selling your soul to the devil before you're even tempted."