The Justice Department told Congress yesterday that it is prepared to challenge the constitutionality of a key section of the Gramm-Rudman-Hollings balanced-budget law.

The department's view, if upheld by the courts, would make the sweeping budget cuts sought by the law no longer automatic. The power to enforce the mandated spending cuts would then revert to Congress, which enacted the law because it has been unable to agree on such cuts in the past.

The department was responding to a suit by Rep. Mike Synar (D-Okla.) that contends that the entire law is unconstitutional. The government's brief sought to have Synar's claims thrown out on procedural grounds, but at the same time served notice that should the case be heard on the merits, it will press the constitutional issue.

The department's brief challenged the role assigned by the legislation to the General Accounting Office, which would have the final word on billions of dollars in budget cuts for the next five years. The GAO role, the department argued, is an unconstitutional infringement on President Reagan's budget-making powers.

Under Gramm-Rudman-Hollings, if the GAO's role is struck down by the courts, both houses of Congress would have to approve the budget cuts in a joint resolution.

Attorney General Edwin Meese III contended in letters to House and Senate leaders yesterday that the comptroller general, as the head of the GAO, "is a legislative officer who may not perform duties that are inconsistent with that status." The department regards the budget rescissions called for under Gramm-Rudman-Hollings as such an executive-branch function.

Senate legal counsel Michael Davidson, who filed a brief supporting the law yesterday, said that the GAO's role "makes 99 percent of the difference. It was very important for Congress to have a procedure in which [budget cuts] would be automatic if they failed to produce a budget. The whole arrangement could stalemate."

The administration has thus gone to court objecting to a central part of the deficit-reduction legislation that President Reagan embraced when Congress passed it 20 days ago.

White House spokesman Larry Speakes said in Palm Springs, Calif., that the administration wants "clarification" on the law's constitutionality.

Congress adopted the law, despite warnings from its own supporters that it might wreak havoc on the budget process, in an attempt to force action on federal budget deficits that could top $200 billion for the rest of the decade. The law, named for Sens. Phil Gramm (R-Tex.), Warren B. Rudman (R-N.H.) and Ernest F. Hollings (D-S.C.), calls for an initial $11.7 billion in budget cuts to take effect March 1, and mandates a balanced budget by fiscal 1991, which begins Oct. 1, 1990.

The law requires the White House Office of Management and Budget and the Congressional Budget Office to calculate program-by-program spending reductions under a complicated formula spelled out in the bill.

Their report would be reviewed by Comptroller General Charles A. Bowsher, head of the GAO, who would certify to the president that the cutbacks would reduce the deficit to levels cited in the law, declining to zero in fiscal 1991. The president would then sign the cutbacks into law.

The law also contains a provision allowing members of Congress to challenge it in court. Synar and 11 other House members filed suit Dec. 19, saying that the statute is an unconstitutional delegation of Congress' power of the purse to the president. Synar's suit, which is being heard by a special three-judge panel, also challenges the GAO's role in the budget-cutting process.

But the Justice Department charged that the provision allowing Synar to sue is itself unconstitutional. The department said that the Constitution requires a particular "case of controversy" to justify a lawsuit, and that a member of Congress has no generalized right to challenge a law as an infringement on his legislative duties or on behalf of his constituents.

The argument over standing is likely to be short-lived. Even if the courts throw out the Synar suit, interest groups affected by the budget cuts are also certain to file suit, as the National Association of Retired Federal Employes has already done.

The crucial legal test, therefore, will involve the separation-of-powers argument. The comptroller general is a presidential appointee, but the Reagan administration contends that the GAO is nevertheless a congressional agency that should be barred from executive-branch duties.

The Justice Department used this argument to challenge a 1984 law that gave the GAO the power to hold up the awarding of federal contracts until it ruled on protests by losing bidders. In a letter to the House last year, the department noted that the comptroller general cannot be fired by the president, only by a joint resolution of Congress.

"The comptroller general is unquestionably part of the legislative branch and is directly accountable to Congress . . . . The comptroller general may not act in an executive capacity, and he may not take actions that bind individuals and institutions outside the legislative branch," the department's letter said. The contracting case is on appeal.

Senate counsel Davidson said in his brief that the GAO's role is crucial to making the legislation work. He told a reporter that the Justice Department's objection is "abstract" because no one has tried to fire the comptroller general since the job was created in 1921.

"He does provide reports to Congress, but he also audits and settles the accounts of executive agencies," Davidson said.

The GAO also defended its budget-cutting role under Gramm-Rudman-Hollings in a brief filed yesterday by the law firm of former White House counsel Lloyd N. Cutler. The House declined to file a brief.

The Justice Department used the same legal logic to argue that the Congressional Budget Office, whose director is appointed by Congress, should play no role in the deficit-reduction law.

The Justice Department believes that Gramm-Rudman-Hollings will work well under a "backup mechanism" in the law, which takes effect if the courts strike down the GAO's role. Under that approach, after the OMB and CBO calculate how much spending would have to be cut to meet deficit targets, both the House and Senate would have to approve the cuts in a joint resolution.

Any impasse, such as a disagreement between the House and Senate or the president's refusal to sign the resolution, would kill the process. Unlike the procedure involving the GAO, the budget cuts are not automatic.

A spokesman for Rudman said yesterday that the senator does not believe the backup approach would produce the same stalemate that has prevented Congress from reducing the federal deficit. He said Rudman believes that even the backup approach "will keep the pressure on Congress" to cut the budget and that it "would take an awful lot of courage" for lawmakers to vote against such a resolution.

A ruling in Synar's suit could come as early as mid-January.