A federal judge ruled yesterday that the House of Representatives can proceed with its historic case against Anne M. Gorsuch, administrator of the Environmental Protection Agency, for contempt of Congress.

U.S. District Court Senior Judge John Lewis Smith Jr. threw out a lawsuit by the Reagan administration claiming that the House could not prosecute Gorsuch because she acted under orders from the president.

The lawsuit was viewed by both sides as a test of the limits of executive privilege, the doctrine invoked by the executive branch to keep sensitive information from Congress and the public. It also posed an unprecedented confrontation: the executive branch of government suing the legislative branch before the judiciary.

In a seven-page ruling Smith said the judiciary should not intervene until Gorsuch becomes a defendant in the criminal contempt proceeding. Smith called on both branches to "settle their differences without further judicial intervention."

Stanley M. Brand, general counsel to the clerk of the House, said the decision was a "total victory" for the House, which cited Gorsuch for contempt in December after she refused to turn over documents to a House subcommittee investigating her agency's efforts to clean up hazardous waste dumps.

Soon after the decision was handed down, Assistant Attorney General J. Paul McGrath offered to negotiate with the House to resolve the dispute over the documents "in a spirit of compromise and cooperation" short of a criminal case.

But the Justice Department, which has 60 days to file an appeal of Smith's ruling, left open the possibility of further appeals, possibly to the Supreme Court.

"We are going to do what the court said," McGrath said. "We are going to try to settle our differences with the House without further involvement of the courts. If we can't, obviously we have to consider where we have to go from there."

Rep. Elliott H. Levitas (D-Ga.), chairman of the House Public Works subcommittee that issued the subpoena to Gorsuch and brought the contempt citation against her, was skeptical of the possibility of compromise.

"But if they are serious and want a compromise," Levitas said, "I'm willing to go the extra mile. The ball is clearly in their court. I am perfectly willing to sit down. But ultimately we have to examine the documents."

Each side said yesterday that it has tried repeatedly to avoid a constitutional confrontation over dozens of EPA documents that Gorsuch, acting under a direct order from President Reagan, refused to turn over.

The Justice Department said that the documents revealed EPA strategies in litigation against polluters and that disclosure of those strategies would jeopardize the cases.

The House maintains that, without the documents, it cannot fulfill its constitutional duty to see that EPA is properly administering the 1980 "superfund" law governing the cleanup of hazardous waste dumps.

The House's last offer, rejected by the White House and the Justice Department, was to allow only subcommittee members and five aides to review the documents in executive session at the EPA or the Justice Department. Under that offer, the House members could take notes but not photocopy the disputed documents.

The Justice Department's last offer was for two EPA lawyers, one of whom would be a career employe, two Justice Department lawyers, one of whom also would be a career appointee, and the White House to screen the documents and keep from the subcommittee only those documents they all agreed would jeopardize pending litigation against polluters. Those documents would be described to the subcommittee, but not turned over.

Levitas said yesterday that the offer was "totally unacceptable."

U.S. Attorney Stanley S. Harris is required by federal law to take the Gorsuch case to a grand jury. Criminal contempt is a misdemeanor carrying a maximum penalty of one year in prison and a $1,000 fine.

Smith's narrow opinion did not deal with the merits of the executive privilege claim, but only with when judicial intervention would be appropriate.