The Justice Department argued for the first time in court papers yesterday that the independent counsel statute under which a host of current and former high Reagan administration officials are being investigated is unconstitutional.

In a brief filed in federal appeals court here, the department said the 1978 Ethics in Government Act should be overturned because it violates the constitutional mandate of separation of powers. The filing came in a case involving an investigation stemming from the Environmental Protection Agency's 1983 Superfund scandal.

Justice Department officials have criticized the statute on constitutional grounds in congressional testimony, but in previous court challenges avoided directly addressing the constitutionality of the law, passed in response to Watergate and the firing of special prosecutor Archibald Cox.

The department's action comes as an unprecedented number of administration officials, including Attorney General Edwin Meese III, are the subject of independent counsel investigations, and as the constitutionality of the law has been challenged by Lt. Col. Oliver L. North and former White House aides Michael K. Deaver and Lyn Nofziger.

In its court brief and in other statements yesterday, the Justice Department asserted that it was firmly committed to investigating wrongdoing by high government officials and that it had taken pains to ensure that the existing independent counsel probes would not be derailed by offering the prosecutors "backup" appointments by Meese.

In letters yesterday to House Speaker Jim Wright (D-Tex.) and Vice President Bush in his role as president of the Senate, Deputy Attorney General Arnold I. Burns said the department filed the brief seeking to have the ethics law declared unconstitutional because it "was compelled to protect the separation of powers that is so fundamental to our constitutional structure."

Burns is acting attorney general in the case because Meese, under scrutiny by independent counsel James C. McKay for his ties to the scandal-ridden Wedtech Corp., recused himself.

Burns said the department is "firmly convinced that the goal of punishing criminal activity by government officials can properly be achieved within the existing constitutional framework" and had "consistently attempted to administer the law in a fashion that avoids its constitutional infirmities."

Fred Wertheimer, president of Common Cause, the lobbying group that has vigorously supported the independent counsel law, called the department's action "outrageous" and "mind-boggling." Cox is now the chairman of Common Cause.

"The bottom line is they are intent on undermining and doing away with the independent counsel statute at the precise time when this administration is under a whole series of investigations," Wertheimer said, warning that the position "could dangerously undermine" all the independent counsel probes.

The brief filed yesterday involved a challenge by Theodore B. Olson, a former assistant attorney general, and two others subpoenaed in the case, former deputy attorneys general Edward C. Schmults and Carol E. Dinkins. The case was referred to in the filing yesterday only as "In Re Sealed Case."

Independent counsel Alexia Morrison is investigating whether Olson gave false testimony to Congress in 1983 during the dispute over the administration's initial refusal to turn over documents concerning the Superfund toxic waste program.

Morrison said yesterday that "we believe the law is constitutional. Chief Judge {Aubrey E. Robinson Jr. of U.S. District Court} has so held and we think that his position is the correct one."

Lawrence E. Walsh, the independent counsel probing the Iran-contra affair, said yesterday he will seek to submit a brief in the Olson case "in support of the constitutionality of the Ethics in Government Act."

Walsh has accepted a backup appointment from Meese, but spokesman James Wieghart said yesterday that Walsh would enter the Olson case because he is "anxious" to have the law upheld and "accepted the parallel appointment only so the investigation could proceed."

McKay, the Wedtech independent counsel who recently obtained an indictment of Nofziger, has also accepted a backup appointment.

Morrison and Whitney North Seymour Jr., the independent counsel prosecuting Deaver, both rejected the parallel appointment.

Morrison said she declined the parallel appointment by Meese because, among other reasons, "Congress enacted the Ethics in Government Act because it believed in these matters the Justice Department should not be involved, and under our circumstances it seemed to us that it would be a direct contravention of that congressional intent to accept a parallel appointment."

In July, Robinson ruled in the Olson case that the independent counsel law is constitutional, the first direct court decision on the constitutional question.

The Justice Department, which decided to enter the case when Robinson's ruling was appealed, argued in its brief yesterday that the ethics act violates the appointments clause of the Constitution by permitting a court, rather than the president, to appoint the independent counsel.

In addition, the department argued, the law violates separation-of-powers provisions by having courts engage in the "nonjudicial functions of determining the scope of an independent counsel's authority, and of deciding when the office of an independent counsel should be terminated because its assigned task is thought to have been completed."

Likewise, the brief said, the law "also contravenes the Constitution by eliminating or strictly limiting the power of the Executive Branch to appoint, control, and remove an officer charged with the quintessential executive duty of criminal law enforcement."