With optimism exceeded only by enthusiasm, Senate Majority Leader Robert J. Dole (R-Kan.) has grabbed center stage on Capitol Hill in a daring bid to accomplish a task that has driven both President Reagan and the Democrats far back into the wings.

In the process, however, Dole has created a test of his own leadership skills less than a month after he assumed the reins of a Republican majority that even senators say has been sorely in need of order, consensus and discipline.

The test is whether Dole can put together a package of spending cuts that exceeds anything passed during Reagan's first term but is still palatable enough to pass Congress.

As some Capitol Hill hands see it, failure could put the brakes on Dole's fast start as Senate leader and slow his drive to win the Republican presidential nomination in 1988, while a victory could establish him as the preeminent man of action in Washington.

A mixed bag of results, they say, could lead to the kind of trench warfare that diminished the command of his predecessor as majority leader, Sen. Howard H. Baker Jr. (R-Tenn.), who retired last year in a mood of considerable frustration.

The goal is to cut deficits by half to less than $100 billion over the next three years, starting with at least $50 billion in spending cuts in the 1986 fiscal year that starts Oct. 1.

That goal was set by Dole and other Senate GOP leaders on Jan. 4 after a meeting in which White House officials -- who originally had proposed the same target figures -- conceded that the goal was beyond their grasp.

Dole also served notice that Senate Republicans would aim to have their plan in hand by Feb. 1, three days before Reagan's budget request is due in Congress. Then he put Senate committees to work on lists for specific spending cuts, which is where the problems started and the momentum stalled.

As committees pondered the translation of glossy goals into painful realities, some met their targets, while others fell short. But the main problem was several days of rhetorical zig-zags from Armed Services Committee Chairman Barry Goldwater (R-Ariz.) that boiled down to strong misgivings about big defense spending cuts.

When Goldwater balked, others got skittish, and then "things began to unravel," according to Sen. Mark Andrews (R-N.D.), a member of the Budget Committee and a strong advocate of deficit reductions.

Goldwater said yesterday that Pentagon officials "can live with" a smaller budget increase than the scaled-down request that Defense Secretary Caspar W. Weinberger negotiated with the White House.

Weinberger reportedly has agreed to ask for an increase of 6.4 percent for 1986, but Goldwater, in an interview on NBC-TV's "Meet the Press," said this could be reduced to about 5.5 percent.

The original strategy was to "bring common sense" on the issue to the White House, Andrews said. When that failed, Dole pursued a "let's-reason-together" strategy among his Senate colleagues, Andrews added. Now "it's the old Bob Dole -- flip, glib, pulling rabbits out of the hat."

But Andrews, like most of his colleagues, believes Dole still may be able to pull it off.

"He has the experience, the knowledge and the skills . . . . He wants a success. Ego can be a weakness, but leadership without ego may not be worth a damn. He's got the skills to back it up," Andrews said of Dole.

Failure to nail down the targeted amount of spending cuts would not necessarily mean failure for the deficit reduction drive.

It could increase pressure for tax increases, as Sen. John H. Chafee (R-R.I.), No. 3 in the Senate GOP hierarchy, has suggested.

But some also worry that the whole effort could fizzle, especially if it gets tangled with Reagan's opposition to tax increases and Democrats' fear of tangling with Reagan on the tax issue. They note that, with the economy growing, inflation relatively low and interest rates falling, it is difficult to create a sense of urgency in a legislative body with a notoriously short attention span.

The core of the Senate plan has been an across-the-board freeze of spending authority at this year's levels, including defense and Social Security benefits, both highly controversial but essential to lock in savings from other programs, according to freeze proponents.

But the freeze would bring in $38 billion next year, or $16 billion short of the $54 billion goal for the year. It would produce two-thirds of the more than $260 billion in spending cuts needed by fiscal 1988.

So some programs, including many that will be recommended for cutbacks or elimination by Reagan, would be targeted for reduction below freeze levels to arrive at the desired totals.

Troubles surfaced at all levels, but especially over Social Security and defense, threatening the whole framework of a freeze.

House Republicans as well as Democrats balked early at eliminating cost-of-living increases for Social Security, although that problem appeared to recede a bit as Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) indicated they would be willing to talk about Social Security adjustments.

This left defense as the main stumbling block at the end of last week, probably the critical one, Dole indicated. Defense cuts are considered essential to win the support of more liberal lawmakers of both parties for reductions in their own favored social welfare programs.

Weinberger continued to resist any major spending cuts, prompting complaints of "stonewalling" from Senate Republicans. By Friday, Dole counterattacked with a blunt warning that refusal by the administration to compromise on defense was jeopardizing the cuts in domestic spending that the administration wants.

Dole clearly still was angling for signs of willingness to deal on the part of the administration, which many Republicans regard as essential to passage of a budget with sizable spending cuts.

"If we go off on our own in the Congress and the president is not on board, then we can't get anywhere. If the president is not going to be for what we finally agree on, it's going to be tough," Dole said.

It was a tacit admission that there are limits to what the Senate Republicans can do, even under the sometimes Houdini-style leadership of Dole.

Among Senate Republicans, there is little talk of trying to force the Democrats out of the wings yet; their turn will come. But there is a strong feeling among the Republicans that if Reagan and Weinberger don't step out soon, the show may close.