The Senate's once-formidable Republican juggernaut ground to a halt yesterday as its frustrated leaders failed to muster a majority for a compromise budget backed by President Reagan and finally sent the whole matter back to a committee.

In a blow to Reagan and the most serious setback so far for the Senate's GOP leadership, Democrats and moderate Republicans combined to defeat, 52 to 48, a 1984 budget that would have preserved Reagan's tax cut program at the cost of high deficits through most of the 1980s.

The Senate then also turned down, 53 to 46, a higher-tax alternative offered by Sen. Slade Gorton (R-Wash.) and backed by other GOP moderates that envisioned limiting this year's scheduled tax cut for those in higher income groups and repealing future inflation adjustments in tax rates.

The tax and spending resolution was then sent back to the Budget Committee under rules that it report out another version by Tuesday.

The Senate votes threw the budget process into disarray and raised the question of whether Congress would fail to enact a budget for the first time since passage of the Budget Control Act in 1974.

In proposing to send the resolution back to committee, Majority Leader Howard H. Baker Jr. (R-Tenn.) held out hopes for a bipartisan plan and said he would try to convince the White House, which contributed to the budget deadlock by adamant opposition to tax increases, that it should agree to a compromise on taxes and other sticking points.

But presidential spokesman Larry Speakes said Wednesday that Reagan preferred no budget resolution at all to a budget that included tax increases, and Reagan, asked yesterday if he agreed, commented that "resolutions are non-binding."

Yesterday's Senate rejection of the Republican leaders' budget, which Reagan had endorsed in qualified terms after his own budget was brushed aside, was the most dramatic evidence so far of collapse of the once almost solid Republican unity behind Reagan's economic program in the Senate.

"For two years, the Senate voted in lockstep to support the Reagan program of big spending for the Pentagon and big tax cuts for the wealthy," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) in a statement after the vote. "Today, the Senate refused to go any further."

O'Neill added: "The Senate's failure to pass a budget is the direct responsibility of the administration it supports. When the Reagan robots go haywire, you do not blame the robots, you blame Reagan . . . . The president prefers chaos to compromise. Apparently, he believes that politics is the art of sabotage."

Baker and other GOP leaders, including Budget Committee Chairman Pete V. Domenici (R-N.M.), had put considerable personal prestige on the line in trying to put together a budget that would preserve Reagan's tax program and most of his proposed military buildup.

But Republican moderates, long-suffering under the power wielded by party conservatives, balked at the high deficits that the leadership plan produced: $192 billion for next year and even higher in future years if no new taxes are enacted.

They came up with a plan of their own with higher taxes and lower deficits, starting with $181 billion next year, and, in effect, checkmated the leadership because they had enough votes to join with the Democrats in defeating the leadership plan.

Then they were thwarted, too. They accepted modifications proposed by Gorton, including less in the way of new taxes and slightly more for defense in 1985, which picked up more Republicans but split the Democrats. So the moderates also lost, although they clearly became more of a power to be reckoned with in the Senate.

Even the leadership proposal, drafted by Domenici with the support of Baker and other GOP leaders, departed significantly from Reagan's blueprint for spending in the 1984 fiscal year that starts Oct. 1.

It pared his proposal for an increase of 10 percent in defense spending after accounting for inflation to 7.5 percent, and added roughly $11 billion more than he wanted for domestic programs he has been trying to cut back over the last 2 1/2 years.

But, because it went along with him on taxes, restricting revenue increases to only about $8 billion over the next two years and merely suggesting that tax increases may be required beyond that, it was vulnerable to charges that it envisioned deficits of $200 billion or more through 1988.

The GOP moderates--led by Sens. John H. Chafee (R.I.), Mark O. Hatfield (Ore.), Charles McC. Mathias Jr. (Md.), Robert T. Stafford (Vt.) and Lowell P. Weicker Jr. (Conn.)--proposed a defense spending increase of 6 percent after inflation for next year, with 5 percent increases thereafter, and tax increases of $70 billion over the next three years, starting with $14 billion next year.

Gorton's proposal would have kept defense spending at 6 percent, with 5.5 percent next year, and put tax increases at $9 billion in 1984, $12 billion in 1985 and $51 billion in 1985.

The big 1985 tax increase was in line with the figure recommended by Reagan as a "standby" increase for that year, but Gorton's proposal would have required legislation this year enacting all tax increases, which was anathema to conservatives.

Gorton's proposal would have produced a deficit of $184 billion next year, declining to $152 billion by 1988, lower than any of the other Senate proposals except the Budget Committee's plan, which had been disavowed by most Republicans.

Gorton predicted last night after the vote that the Budget Committee, of which he is a member, will wind up accepting a plan that is close to what he offered.

Domenici also indicated that the committee may have to agree to more taxes than the leadership proposed, but Republican conservatives appeared to remain dug in against tax increases, and leadership aides said compromise would be difficult.

The Democratic-controlled House has already approved a budget resolution that includes $30 billion in new taxes for next year, a 5 percent defense spending increase and more than $30 billion more in domestic spending than Reagan wants. The House budget contains a deficit of $174.5 billion.

If Congress passes no budget this year, taxes and spending bills would proceed through the two houses without regard to benchmarks set by the budget or various enforcement mechanisms to keep deficits in line with budget projections.

But Baker, Domenici and other Republican leaders, joined by Sen. Lawton Chiles (D-Fla.), ranking Democrat on the Budget Committee, insisted yesterday that they intended to pass a budget.