A Pentagon commission formed in the wake of the Walker spy case recommends expanding the use of lie detector tests, paying informants who report suspected spies, and authorizing the Defense Department to cooperate with the Federal Bureau of Investigation as part of a broad attack on espionage within the government.

"In general," the commission report yesterday concluded, "the Department of Defense security program has been reasonably effective" but "falls short of providing as much assurance as it might" against spies.

Defense Secretary Caspar W. Weinberger appointed the panel, headed by retired Army general Richard G. Stilwell, in June following disclosures about the alleged sale of military secrets to the Soviet Union by former Navy communications specialist John A. Walker Jr., his son and his brother.

At a Pentagon news conference yesterday, Stilwell said the commission favored a wider use of lie detectors to counter security breaches, but cautioned that polygraphs should be considered investigative tools and not proof of guilt or innocence.

The effectiveness of polygraphs is hotly contested, although Congress already has authorized the Pentagon to administer thousands of tests to officials having access to highly classified information. John F. Beary III, a physician and former assistant defense secretary, has assailed lie detectors "as unsound as the use of Laetrile to cure cancer or the use of a copper bracelet to cure arthritis." He predicted that expanded use of lie detectors would invite lawsuits from employes dismissed on the basis of test results.

Stilwell disputed Beary's views yesterday and described him as "unlettered in the polygraph field." The commission report asks Congress to give the defense secretary authority "to develop a coherent and gradually expanding program" for administering polygraphs to military and civilian employes with access to secrets.

In discussing payments to informants, the commission recommended that money go to Pentagon and contractor personnel who provide information that leads to arrests of spies or helps identify hostile intelligence agents.

Asked if this might generate a bounty-hunting atmosphere within the military, Stilwell compared the recommendation to "crime stopper" programs that he said civilian police departments have used successfully.

In a recommendation that could provoke congressional opposition, the commission said the Pentagon should "explore" with the FBI and Justice Department allowing Defense Department counterintelligence specialists to help in civilian counterintelligence efforts.

The report said the Pentagon "potentially" could provide not only agents but "technical and logistical assets," the latter presumably referring to military aircraft and helicopters.

The commission also recommended what it called "behavioral science research" to determine if drinking and other habits provide clues as to whether a Pentagon employe should be trusted with secrets.

The report said the government should treat "all nations which present a hostile intelligence threat the same way U.S. officials are treated in those countries."

The commission conceded that stemming the illicit flow of classified information is a monumental challenge. In 1984, the commission said, the Pentagon classified 16 million documents, and there could be as many as 100 million others already in the files.

Weinberger, in releasing the commission report, ordered what he called a "one time, top-to-bottom" inspection throughout the military to see how secret documents are kept, who has access to them and whether commanders throughout the chain of command understand Pentagon security procedures.

The commission called for stiffer criminal laws to punish government employes and civilian defense contractors who improperly provide secret information. There are currently 2.6 million people with access to some classified information, the commission said.