Sen. Dan Quayle (R-Ind.) is one of those hard-core Republican loyalists on whom President Reagan and GOP leaders are relying to pass their deficit-reduction compromise in the Senate.

But, good soldier that he is, Quayle has his limits. If the budget battle that opens Monday on the Senate floor turns into a "demolition derby," as Quayle and others say they fear that it will, the junior senator from Indiana plans to opt out.

The problem for Reagan and Senate leaders is to prevent any unraveling of the budget package, for, once started, the unraveling could be impossible to halt, especially if senators like Quayle start going their own way.

At that point, if not before, the plan that has been painstakingly sewn together over the past 3 1/2 months to cut the government's $200 billion-plus annual deficits in half over the next three years almost surely would fall apart.

The difficulty with the Republican package is that, with Reagan having ruled out tax increases or a major slowdown in his defense buildup, the proposed cuts in domestic spending are so deep and wide-ranging that they hit virtually everyone's pet cause.

As many as 17 programs, from the Small Business Administration to Amtrak subsidies, would be eliminated or phased out and nearly 30 would be cut substantially, including such big ones as Medicare and farm-price supports. Even that most holy of sacred cows, Social Security, would be affected; cost-of-living increases for government pensions would be cut roughly by half over the next three years.

At the same time, defense spending would increase by 3 percent after accounting for inflation, an increase that many Republicans as well as Democrats say is excessive in light of earlier defense spending increases and the continued squeezing of domestic spending.

"This is not shared sacrifice," said Sen. David L. Boren (D-Okla.), one of the conservative Democrats who normally would be counted upon by the Republican leadership to offset the defection of liberal-to-moderate Republicans on key budget issues.

Boren, for instance, said that he could support limits on inflation adjustments for Social Security, but only if low-income retirees were protected and defense spending were reduced.

What is working in favor of the White House and Senate Republican leaders is the difficulty of assembling an alternative that will not run into a buzz saw of opposition from Reagan.

And, without an acceptable alternative, pressure will be strong to go along with the Republican plan as the only means of expressing courage on the deficit issue.

The problem with most alternatives that have been discussed is that, in order to restore some of the domestic spending that the Republican plan would cut and still trim deficits by the same amount, they run afoul of Reagan on taxes or defense spending.

Sen. Lawton Chiles (D-Fla.), ranking Democrat on the Senate Budget Committee and principal author of a budget compromise that passed the Senate last year, is developing an alternative that would spread the burden of spending cuts more evenly.

But the Chiles plan will propose both tax increases and defense spending cuts that Reagan opposes, and Democrats are fearful of tangling with Reagan on taxes after Walter F. Mondale's unpleasant experience on that score during last year's presidential campaign. Reagan has dared Congress to "make my day" by proposing tax increases. Few Democrats appear likely to do so unless they have enough Republican company to shield themselves from presidential repercussions.

As of late last week, it was unclear whether the Republican plan could be kept intact.

Democrats sent clear signals that they will move quickly to test Republican discipline on such sensitive spending issues as Social Security. And some Republicans indicated that they will vote selectively with the Democrats on issues such as Social Security.

Senate Majority Leader Robert J. Dole (R-Kan.) claimed at least 35 to 40 hard-core Republican votes for the plan but, despite a week-long series of private conversations with small groups of senators, efforts to nail down swing votes on critical issues appeared inconclusive.

Nor was it clear how the pieces can be put back together if the package falls apart, although some Democrats as well as Republicans were suggesting that disintegration of the GOP plan might precipitate an effort to put together a bipartisan package that could pass.

"It'll be . . . tough," Assistant Senate Majority Leader Alan K. Simpson (R-Wyo.), said of the effort to hold the GOP package together.

"The real question is whether, after the demolition derby is over, there are any cars out there that will start," said Quayle, speaking of what might be done if the plan falls apart.

In a sense, Quayle exemplifies both the strength and weakness of the Senate leadership's position as the battle begins.

A fiscally conservative member of the Budget Committee and a participant in the negotiations that led up to the compromise, Quayle said he intends to swallow his objections to key elements of the package and vote against all amendments, in hopes of keeping it intact.

He plans to vote to limit inflation adjustments for Social Security benefits, although he said that he thinks it's political suicide for Republicans to propose such a move. He said he probably will vote to kill the Amtrak rail passenger service, even though there is a big Amtrak maintenance facility in Indianapolis.

But if the plan starts disintegrating because of amendments on the Senate floor, Quayle said in an interview last week, he will feel free to start voting for modifications.

Although he is in less trouble than most, Quayle is one of 22 Republican senators who are up for reelection next year and he has committed himself to the 1,300 employes of the maintainenance facility in Indianapolis to fight to keep Amtrak alive in some form.

So voting against an expected amendment to restore at least some of the $740 million Amtrak subsidy will be difficult for Quayle. He said last week that, although he was inclined to oppose all amendments, he was still having trouble on the Amtrak issue.

A major imponderable as the voting draws near is the intensity of the lobbying campaigns, pro and con.

While pressure from constituency groups appears uneven and generally no greater than it has been in past budget debates, there were signs late last week that protests are beginning to pick up. And Reagan is expected to turn his attention to the budget once the Senate and House complete action on aid to antigovernment rebels in Nicaragua, presumably by Tuesday.

Dole has been pressing the White House to have Reagan make a televised speech on behalf of the deficit-reduction plan. Such an address is considered likely next week.

"Reagan holds the key," said a Senate Republican aide. "Bad as things look now, he could pull it out if he really makes the bells ring."