The Defense Department announced yesterday that it will widen its use of lie detectors, for the first time authorizing polygraph examinations to determine the "trustworthiness, patriotism and integrity" of non-intelligence employes who might have access to secret information.

The new policy also allows the Pentagon, for the first time, to deny certain positions to employes solely on the basis of their failure to pass a lie-detector test. In the past, because of the unreliability of polygraphs, Pentagon rules said no adverse action could be taken against anyone who had failed a test unless subsequent investigations turned up corroborating evidence.

Retired Gen. Richard G. Stilwell, deputy undersecretary of defense for policy, said that the new policy will apply to 3,500 people this year under a test program approved by Congress. If Congress agrees, it would be expanded to about 10,000 people each year, he said, to help keep spies from infiltrating sensitive military programs.

"Over the past decade, there has been an increase in the hostile intelligence presence in the United States," Stilwell said.

But Rep. Jack Brooks (D-Tex.), chairman of the House Government Operations Committee, said there is "no scientifically acceptable evidence" that polygraphs are effective except in narrowly defined investigations. He added that there is "good reason to believe that they result in high error rates causing harm to many innocent people, the government and national security."

Brooks introduced legislation yesterday that would prohibit the use of lie detectors for screening prospective government workers, as the new Pentagon policy allows.

"Use of polygraphs may create a false sense of security and actually weaken our national defense," Brooks said, adding that in some cases "a coin toss would be more accurate."

Under the Reagan administration, the Pentagon already has expanded the use of lie-detector tests, from about 13,000 in 1980 to 21,000 in 1983, the most recent year for which information is available. All of those tests were conducted under regulations that have been in place since 1975, which allow polygraphs in criminal investigations, in screening prospective employes of the top-secret National Security Agency and in chasing news leaks.

The new regulations allow the Pentagon to expand procedures that have been used to screen NSA and CIA employes to hundreds of other secret activities within the Defense Department. Civilians, military personnel and contractor employes would be affected, Stilwell said.

Stilwell said he could not estimate how many people would be subject to "random" polygraph tests under the new policy, but he estimated fewer than 48,000 would be involved. Brooks said more than 100,000 people in the Defense Department have access to highly classified information and could be affected by the new order.

Stilwell, who is in charge of tracking down security leaks for the Pentagon, had proposed wider use of lie detectors in 1982. Several months later, President Reagan issued an administration-wide directive that would have covered as many as 2.5 million employes and would have required them to take lie detectors in the course of leak investigations.

Both orders prompted criticism from Congress, the press and some medical and law enforcement experts. John F. Beary III, then acting assistant secretary of defense for health affairs, opposed the new policy, telling Defense Secretary Caspar W. Weinberger in a memo that the polygraph "misclassifies innocent people as liars."

Reagan suspended his directive early last year, but the Pentagon won permission in the 1985 defense authorization bill for a one-year "test" of as many as 3,500 polygraphs. The Pentagon will report back to the armed services committees at the end of the year.

The new directive, signed by Deputy Secretary William H. Taft IV, says that people who reveal "deception" on a polygraph may be denied a job on that basis alone if a senior official determines that "the information in question is of such extreme sensitivity that access under the circumstances poses an unacceptable risk to the national security." People denied a promotion on that basis, however, could retain their current jobs or jobs at an equivalent grade.

Stilwell said he believes that polygraphs are accurate between 75 and 90 percent of the time.