A special federal judicial panel yesterday struck down the heart of the Gramm-Rudman-Hollings budget-balancing law, ruling that its automatic budget-cutting provision is unconstitutional.

The unanimous three-judge panel said the law, a drastic and unprecedented effort to control mounting federal deficits, violates the Constitution's separation of powers requirements by giving executive branch decision-making authority to a legislative branch officer, the comptroller general.

The judges stayed the effect of their ruling until the Supreme Court reviews it. Thus, it leaves undisturbed the $11.7 billion in fiscal 1986 spending cuts scheduled for March 1. But the decision, and the unanimity of the panel, reinforced speculation that the Supreme Court will find the law constitutionally flawed, as many in Congress said it was even as they passed Gramm-Rudman-Hollings.

Attorneys for the Senate and the comptroller general said they would appeal the ruling to the Supreme Court. The high court's calendar for this term is full, but sources yesterday said if the court agreed to hear the appeal, it would likely do so in April or early May, with a ruling expected before early July.

Yesterday's ruling, if upheld by the Supreme Court, undercuts the law's power to force Congress and the president to agree to specific and politically painful budget cuts. "Our holding today eliminates the automatic deficit-reduction process," the panel said, and allows budget cuts to be made only in the usual manner, by "legislation" passed by Congress and signed by the president.

"The unique aspect of Gramm-Rudman, the automatic trigger, has been disarmed," said Steven R. Ross, general counsel for the clerk of the House, which, along with the Senate, had asked the panel to uphold the law.

"Having disarmed Gramm-Rudman," Ross said, "you are back to the political process of reaching day-by-day agreement on budget matters, there is no longer a gun pointed at their [lawmakers'] heads."

President Reagan yesterday said the court action did not invalidate the requirement to reach a balanced budget, and he urged Congress to meet the deficit targets contained in the law regardless of court rulings. On Capitol Hill, key legislators said political pressure could compel them to agree on spending cutbacks even without the threat of the automatic trigger ruled out by the court.

"There's no turning back now," said Sen. Ernest F. Hollings (D-S.C.), one of the law's sponsors.

"We ought to operate on the assumption that we have to meet those deficit targets or better them," said Rep. Mike Synar (D-Okla.), who challenged the law with 11 other members of Congress and the National Treasury Employees Union.

The law, enacted in December, required that budget deficits be reduced gradually to zero in five phases by fiscal year 1991. If Congress fails to meet the prescribed targets in any year, the law would have required automatic, across-the-board spending cuts (or tax increases) sufficient to meet the targets.

Three agencies -- the Office of Management and Budget, the Congressional Budget Office and the General Accounting Office, which is headed by the comptroller general, would determine whether Congress had met the target and, if not, how much spending would have to be cut.

Gramm-Rudman-Hollings had barely become law when Synar sued to have the trigger mechanism declared unconstitutional.

Appeals court Judge Antonin Scalia, a Reagan appointee; U.S. District Court Senior Judge Oliver Gasch, an appointee of Lyndon B. Johnson, and District Judge Norma Holloway Johnson, an appointee of Jimmy Carter, formed the panel that ruled yesterday.

The panel said Congress had "established an intricate administrative mechanism to address" the goal of a balanced budget by fiscal 1991 "and it has specified in meticulous detail which program budgets will be reduced in order to achieve that result, and by how much."

The panel said Congress did not give away too much power, as some had argued. It simply gave it to the wrong person, the court said, the comptroller general, who, though appointed by the president, is in effect under the control of Congress because it alone has the power to remove him.

"The grant of authority to the comptroller general was a carefully considered protection against what the House conceived would be the pro-executive bias of the OMB [Office of Management and Budget]," the panel said. "It is doubtful that the automatic deficit-reduction process would have passed without such protection, and doubtful that the protection would have been considered present" if the comptroller were not under congressional control.

"We hold, therefore, that since the powers conferred upon the comptroller general as part of the automatic deficit-reduction process are executive powers, which cannot constitutionally be exercised by an officer removable by Congress, those powers cannot be exercised and therefore the automatic deficit-reduction process to which they are central cannot be implemented."

The panel, departing from the strict legalistic tone of the first 48 pages of its 50-page opinion, said it did not mean to "minimize the effect of our invalidation of one small section of the act upon the entire statutory scheme . . . .

"It may seem odd that this curtailment of such an important and hard-fought legislative program should hinge upon the relative technicality of" who has the power to remove the comptroller general.

"But the balance of separated powers established by the Constitution consists precisely of a series of technical provisions that are more important to liberty than superficially appears, and whose observance cannot be approved or rejected by the courts as the times seem to require."

Congress, concerned that the automatic trigger might be unconstitutional, included a "fallback" procedure, under which the OMB and Congressional Budget Office would still calculate how much spending would have to be cut to meet deficit targets. The panel said it "saw no reason to doubt" the constitutionality of the fallback.

But, under the fallback, both the House and Senate would have to approve the cuts in a joint resolution. If either the House or the Senate refused to pass the resolution or if the president refused to sign it, the process would come to a halt.

Ross said House rules may not even require a "straight up or down vote on the resolution. It may be subject to amendment, in which case it is not unlike any budget resolution."

"Everyone was willing to agree to" the targets, Ross said. "It has always been the specific cuts" that caused budget deadlock.

The panel's opinion, signed by all three judges, was based on reasoning many observers say is most likely to be upheld by the Supreme Court, which has been in recent years particularly sensitive to constitutional separation of powers questions.

The court in 1976 struck down the appointment procedures for some members of the Federal Election Commission because they were to be appointed by Congress to perform executive branch functions. In 1983, the justices struck down a legislative veto scheme in which one house of Congress could overturn executive branch actions and, in effect, legislate outside the Constitution's strick guidelines for enforcing laws.