The Justice Department appeared in a brief filed this week to ally itself with conservatives who argue that a host of federal regulatory agencies are operating unconstitutionally, legal experts said yesterday.

The separation-of-powers debate surfaced anew in a lawsuit challenging the constitutionality of the Gramm-Rudman-Hollings balanced-budget law. The Justice Department said Tuesday it is prepared to argue that the General Accounting Office cannot play a role in mandating budget cuts as the legislation requires because that is an executive branch function.

The GAO, in the Justice Department's view, is not part of the executive branch, in part because its director cannot be fired by the president.

The department's former chief legal adviser, Theodore B. Olson, has been making a similar argument in a suit challenging the enforcement authority of the Federal Trade Commission.

Olson, representing a mortgage company accused of price fixing, contends that the FTC has no power to enforce the law because its commisisoners cannot be fired by the president and, thus, are not really part of the executive branch.

This attack on the FTC, if upheld by the courts, could strip enforcement powers from more than a dozen independent regulatory agencies whose members are also not routinely removable by the president, as are all other executive branch political appointees.

Among the agencies are the Securities and Exchange Commission, Federal Communications Commission, Consumer Product Safety Commission and Interstate Commerce Commission.

Attorney General Edwin Meese III appeared to embrace this view in a speech in September to the Federal Bar Association. "The real law-making power in Washington is wielded . . . by relatively anonymous members of the federal agencies," he said, adding that these officials are accountable to neither the president nor Congress.

"It should be up to the president to enforce the law . . . We should abandon the idea that there are such things as 'quasi-legislative' or 'quasi-judicial' functions that can be properly delegated to independent agencies," Meese said.

Justice Department officials have not decided whether to make this argument in court. They have not yet filed a brief on the merits in the FTC case.

The Gramm-Rudman-Hollings suit, filed by Rep. Mike Synar (D-Okla.) and 11 other House members, has also begun with procedural sparring, with the administration saying that the lawmakers have suffered no injury and, thus, have no legal standing to sue.

That "standing" argument probably will not apply to the National Treasury Employes Union, which filed suit yesterday challenging the law. Its retired members had their pension benefits frozen Jan. 1 under Gramm-Rudman-Hollings.

Should the Synar case be heard, the Justice Department has said it would oppose the GAO's role as the final arbiter on the billions of dollars in budget cuts needed to balance the budget by fiscal 1991. Justice says giving the GAO such a role would be an unconstitutional infringement on the president's budget-making authority.

The GAO disagreed in a brief filed by former White House counsel Lloyd N. Cutler. "If the Justice Department is right" about restrictions on his office's authority, said GAO general counsel Harry R. Van Cleave, "it would be equally true of the commissioners of the regulatory agencies."

The GAO brief said that the 1921 law setting up the office called for an independent comptroller general, "to insulate his important watchdog functions from political pressures."

That law, which transferred auditing authority held by the Treasury Department since 1789, provided that the comptroller general could be fired only by a joint resolution of Congress signed by the president. (An earlier version giving Congress sole right of dismissal was vetoed by President Woodrow Wilson.)

Members of independent regulatory agencies also cannot be fired by the president, except for cause.

The Supreme Court upheld this principle in 1935, ruling that President Franklin D. Roosevelt could not dismiss an FTC commissioner for personal or political reasons.

House counsel Steven R. Ross said the Justice Department is "moving down the same path" in the cases involving the GAO and the FTC.

Ross said he opposes the view "that the independent regulatory agencies should be made non-independent," saying that Congress intentionally designed them to be free from White House pressure.

But Olson, who headed the Justice Department's Office of Legal Counsel during President Reagan's first term, said such agencies cannot have prosecutorial power unless they are under the day-to-day control of the president.

Olson, now a law partner of former attorney general William French Smith, is representing Ticor Title Insurance Co., one of six title-insurance firms accused of fixing prices by the FTC.

Other experts cautioned that the GAO's status is unique within the government and that the administration has little incentive to rein in the regulatory agencies now that Reagan has appointed a majority of their members.

"Whatever their merits, there certainly is a distinction between the two cases, because the GAO is an arm of Congress while the Federal Trade Commission is not," said former solicitor general Rex E. Lee