Senate Republican leaders got off to a shaky start yesterday in their push for a quick symbolic victory in the battle over deficit reductions as they backed away from a showdown to avoid defeat at the hands of Democrats and GOP dissenters.

The setback occurred near the end of the first day of long-delayed debate on the Republicans' plan to make deep new cuts in social spending in order to halve the government's $200 billion budget deficits over the next three years.

With the outcome apparently hinging on a razor-thin margin, Majority Leader Robert J. Dole (R-Kan.) first pushed for a quick vote but was blocked by delaying tactics from Minority Leader Robert C. Byrd (D-W.Va.).

Then Byrd, in a reversal of the cat-and-mouse game that he and Dole had played all day, made a sudden shift last night and agreed to an early vote, apparently sensing that support had shifted his way.

This threw Republicans off guard and, after several huddles on and off the Senate floor, ultimately forced Dole to recess the Senate for the evening. Reflecting partisan lines sharply drawn from the start, the vote to recess was 52 to 44, along party lines.

"We're going to try to turn it around tomorrow," Dole said. "If it takes a month to turn it around, we'll take a month."

Acknowledging that he was not sure if he had the votes to give the plan even tentative approval, Dole said, "The point here is winning. We may lose eventually, but we're going to do a lot of work first."

As the day began, Republicans hoped to capitalize on President Reagan's nationally televised speech Wednesday night on behalf of the plan by forcing a preliminary vote that, if successful, could increase momentum for the plan and possibly help fend off crippling amendments later.

"We have to demonstrate up front that we're serious about deficit reduction," said Dole, employing an intricate parliamentary procedure to overcome Democratic resistance to a quick vote.

Democrats responded by serving notice that they would attempt a parliamentary ploy to force even earlier test votes on such flash-point issues as cutbacks in cost-of-living increases for Social Security benefits.

Dole said at the outset that the process would be "tortuous" and was quickly proven correct.

Within hours, the chamber virtually had emptied as Democrats, stalling, forced the Senate clerk to read almost all 55 pages of a procedural amendment from Republicans.

"We will not be rushed," Byrd asserted at one point, only a couple of hours before he switched tactics and agreed to vote within 30 minutes, prompting a scramble by Republicans to round up colleagues who had wandered out to dinner.

The fight is over a compromise budget negotiated by the White House and Senate Republican leaders to reduce deficits by nearly $300 billion over the next three years, largely by making deep new cuts in domestic spending.

It would eliminate or phase out nearly 20 programs, ranging from Amtrak subsidies to small business loans. It would make deep cuts in several dozen other programs, such as Medicare, student loans and farm supports. It would freeze spending for most of the rest, including that of the politically sensitive Social Security retirement system.

It also would trim Reagan's defense budget, although military spending would be allowed to rise 3 percent above inflation. No tax increases are proposed.

By cutting $52 billion in spending next fiscal year, the plan would produce a deficit of $175 billion for fiscal 1986, declining to $98 billion by fiscal 1988 and possibly to a balanced budget by the decade's end, according to Republicans.

Embracing the deficit-reduction cause as fervently as the Republicans, Democrats agree that deficits should be reduced, with more emphasis on spending cuts than tax increases. But they say the Republican plan favors rich over poor and military over domestic spending.

"In the past several years, the budget has become an instrument of national retrenchment rather than an investment in our future," said Sen. Lawton Chiles (Fla.), ranking Democrat on the Budget Committee.

"The budget today is now used almost exclusively to cut spending without a real connection to the kind of country we want to build and without a sense of the role the federal government should play in that effort," he said.

As a group, Democrats have presented no alternative, although they plan a series of efforts to restore spending that Republicans would cut from specific programs.

In addition, several individual Democrats have comprehensive alternatives. The two major ones, different proposals advanced by Chiles and Sen. Ernest F. Hollings (D-S.C.), including tax increases.

Throughout most of the last two days, Dole lobbied Republican colleagues for support on the first test vote, assuring them that they would have ample opportunity on later votes to change elements of the package.

In addition, Reagan's speech prompted a flood of phone calls to senators' offices, and the president was reportedly lobbying senators by telephone during the day.

With no assurance of any Democratic support, Dole could afford to lose no more than two or three Republican votes, and many more than that have objected strongly to key items.

Even before debate began, Dole was testy in expressing impatience with senators who make speeches about reducing deficits and then balk at specific proposals.

"I don't want to hear any more deficit-reduction speeches from those who stand up and vote against us for the next 10 days," he told reporters. "I'm tired of it. I've heard all the deficit speeches I can tolerate from guys who stand up and talk about the deficit and then don't want to do anything about it. They want to raise your taxes."