House Democratic leaders, challenging a budget "compromise" reached last week by President Reagan and Senate Republicans, yesterday unveiled an alternative that would raise more in taxes, shave more from defense, cut less in domestic spending and leave Social Security untouched.

The Democrats' counterproposal, advanced by House Budget Committee Chairman James R. Jones (D-Okla.) with the qualified support of House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), came as Senate Republican leaders ran into trouble over their own plan to help reduce budget deficits in part by cutting $40 billion from Social Security over the next three years.

As of last night, Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) was considering shelving the administration's huge defense authorization bill because of a rider, which Democrats were attempting to attach, that would force senators to go on the record as for or against cuts in Social Security.

Some Republican moderates were siding with the Democrats and the vote looked "very close," Baker conceded earlier in the day. But Senate Armed Services Committee Chairman John G. Tower (R-Tex.) was understood to want to go ahead with the measure today, leaving the bill's standing in doubt.

A vote by the Republican-controlled Senate against the Social Security provisions of Reagan's latest budget plan would be an embarrassing setback, especially at a time when House Democrats were rolling out an alternative that avoided Social Security cuts and still pointed toward a lower fiscal 1983 deficit than the Reagan-blessed plan--$102.9 billion as opposed to $106.1 billion.

Moreover, the Jones plan got a friendly nod from Rep. Bill Green (R-N.Y.), co-chairman of the moderate Republican "Gypsy Moths" in the House, some of whom joined with moderate Democrats last week in floating a similar plan as the foundation for a bipartisan budget.

"This is better than the Senate plan," Green was quoted by an aide as saying. It's "significantly closer" to the Moths' own plan than the one adopted last week by the Senate Budget Committee and endorsed by Reagan, Green added.

Jones' plan, which the House Budget Committee is scheduled to begin working on Wednesday, has these key elements:

A three-year tax increase of $145 billion, compared with the Senate plan for $95 billion in tax increases over three years. Jones said his plan would leave open the question of modifying Reagan's 10 percent individual income tax cut scheduled for July, 1983, which the president has insisted on preserving. Next year's tax increase would be $30 billion under Jones' plan and $20 billion under the Senate plan.

A freeze for three years in domestic appropriations that is somewhat less restrictive than the Senate proposal, saving $32.7 billion instead of $38.3 billion. An extra $950 million in domestic spending would be allowed for programs such as job training, student loans, the space program and crop insurance--both easing the effects of earlier program cuts and helping to pick up votes of moderate Republicans as well as the conservative Democrats who supported Reagan's budget last year.

Cuts in Reagan's big defense buildup program amounting to $46.2 billion over three years, compared with $22 billion under the Senate plan. Jones' proposal would allow for a 5 percent "real," or after inflation, growth in defense spending. The Senate plan anticipates real growth of about 7 percent.

A 4 percent increase in federal pay and pensions over the next three years, compared with a freeze next year, followed by 4 percent increases in the two following years, under the Senate plan.

Cuts in basic benefit or entitlement programs, other than Social Security, totaling $14.3 billion over the next three years, as opposed to the $33.6 billion in cuts recommended by the Senate committee. For next year, the figures would be $3 billion and $7.2 billion, respectively.

No "shot in the dark on what is needed for Social Security," as Jones put it. Unlike Jones' plan, the Senate committee proposed $6 billion in Social Security cuts next year, for a total of $40 billion by 1985, with details to be developed by a commission that is currently studying the system's solvency problems.

In all, the Jones plan anticipates deficits of $102.9 billion for 1983, $70.8 billion for 1984 and $31.1 billion for 1985. The figures under the Senate plan are $106.1 billion, $69 billion and $39.5 billion, respectively.

Neither Jones' plan nor the Senate Budget Committee's proposal make specific recommendations for tax increases or program-by-program spending cuts. These would be left up to individual authorizing and appropriating committees under the general targets of the budget resolutions that each house will adopt and then attempt to reconcile in a House-Senate conference.

However, Jones' proposal for $950 million in domestic appropriations "add-ons" appeared to be an attempt to lure lawmakers with specific needs--or to address "specific pressure points," as one aide put it. Details were still being worked out, aides said.

As for the 1983 tax cut, which Reagan is determined to protect and the Senate plan effectively shields, Jones said it is "on the table for consideration"--neither ruled in nor ruled out.

While O'Neill said he was "generally supportive" of Jones' plan, he was also quoted by an aide as indicating he would be reluctant to compromise any further. "These are bottom-line figures as far as Democrats are concerned," O'Neill was quoted as telling his aides.

Jones claimed that House Democratic leaders, along with Republican Gypsy Moths and conservative Democratic "Boll Weevils" that he has talked to, agree with the general outlines of the plan. It is aimed, he added, at "the broad middle of both parties, no question about it."