The Senate yesterday approved legislation to overcome Supreme Court objections to automatic enforcement rules in the Gramm-Rudman-Hollings budget law after reaching a bipartisan compromise that would limit proposed new presidential powers to order spending cuts to meet deficit targets.

The Republican-controlled Senate approved the plan, 63 to 36, as part of a bill to extend the federal debt ceiling by $244 billion to $2.323 trillion. The measure must be passed before Congress recesses in two weeks for the rest of the summer.

Fate of the budget enforcement plan is unclear in the Democratic-controlled House, which will be torn between pressures for prompt passage of the debt-ceiling measure and strong political opposition to expansion of presidential powers.

The plan empowers the president's Office of Management and Budget to implement spending cuts required by Gramm-Rudman-Hollings but constrains its discretion by spelling out economic assumptions and other key factors for next year and giving Congress authority to do so again in future years if it disagrees with OMB's decisions.

The 10-day stalemate over how to fix the budget-balancing law was broken as both the House and the Senate ran into trouble meeting deficit-reduction targets aimed at avoiding the automatic, across-the-board spending cuts that Gramm-Rudman-Hollings requires if targets are not met.

For different reasons, the two chambers were between $1 billion and $2 billion short of the $9.2 billion goal in budget "reconciliation" legislation to comply with the fiscal 1987 budget.

In addition, the Senate Finance Committee was wobbling over its earlier decision to increase the federal cigarette tax by 8 cents a pack, raising the prospect of other taxes or a bigger shortfall on the deficit.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) vowed that the House would meet the target one way or another, although how was not immediately clear. The Senate would probably have to decide ultimately whether to insist on full compliance or settle for something short of it, said Senate Budget Committee Chairman Pete V. Domenici (R-N.M.).

The uncertainty underscored expectations that Congress would miss its deficit target for fiscal 1987 by about $20 billion, according to sources. Before counting savings currently under consideration, Congressional Budget Office officials are tentatively anticipating a deficit of about $175 billion, sources said.

But other budget experts said Congress could come within striking distance of avoiding spending-cut sanctions if it approves all, or nearly all, of its proposed budget savings. The deficit target for next year is $144 billion, and spending cuts are required if the target is exceeded by $10 billion or more.

Settlement of the political, fiscal and institutional controversy over whose finger will be on the Gramm-Rudman-Hollings "trigger" was expected to remove at least one major obstacle to compliance with the complicated new budget law.

Without some automatic enforcement provisions, many lawmakers have been skeptical that Congress would cut deficits enough to comply with the law.

As enacted by Congress last year, the law set deficit targets to achieve a balanced budget over five fiscal years, and required the General Accounting Office to impose across-the-board spending cuts to meet the targets if Congress and the president failed to do so.

But the Supreme Court earlier this month held that GAO was an agency of the legislative branch and could not exercise executive powers to carry out the cuts, thereby tossing responsibility for making the cuts back to an unwilling Congress, where senators immediately set about trying to repair the automatic trigger.

Sens. Phil Gramm (R-Tex.), Warren B. Rudman (R-N.H.) and Ernest F. Hollings (D-S.C.) proposed to empower the president's Office of Management and Budget to enforce the law, but Democrats objected and a compromise was worked out between Domenici and Sen. Lawton Chiles (D-Fla.), ranking minority member on the Budget Committee.

The Domenici-Chiles compromise attempts to avoid constitutional objections by writing as much as possible into law, including economic assumptions upon which deficit projections are made, and many of the factors governing allocation of spending cuts, such as defense "spend-out" rates and specifications for sales of government assets.

For fiscal 1987, the economic assumptions would be an average of estimates by OMB and CBO. For economic growth, for instance, Congress would write into the law a figure splitting the difference between OMB's anticipated growth rate of 3.8 percent and CBO's expected estimate of 3.2 percent. On most other economic indicators, the two estimates are expected to be closer.

For future years, the House and Senate Budget committees could propose -- and Congress enact -- their own economic estimates if they disagree with OMB's final verdict. If the changes were not enacted and signed by the president, or not passed over his veto, OMB's decision would stand.

While proponents contended it would meet constitutional tests, others were less sure. It was, said Sen. Daniel Patrick Moynihan (D-N.Y.), "fraught with constitutional peril." Others, including Sen. Gary Hart (D-Colo.), contended it invited OMB exploitation. "The fix has not yet fixed the problem of OMB manipulation of the economics," Hart said.

In the vote, Republicans split 42 to 10 in favor of the OMB proposal; Democrats divided with 26 against it and 21 for it. Virginia senators voted for it, Maryland senators against it.