WHEN THE DEPARTMENT of Justice announced it was beginning an internal inquiry into who leaked details of the Abscam investigation to the press, one senior attorney there is said to have described it as a "fool's errand." No one was going to find out anything. But that was not what Attorney General Benjamin R. Civiletti had in mind. He really wants to know who leaked, and he wants to know badly enough to have authorized his investigators to use the lie-detecting machine.

Some employees of the department are reported to have been asked already to submit to this test for truthfullness. Others may be asked. Asked, however, is not quite the right word because, as the attorney general explained the department's rules, there is a degree of "compulsion" in such a request.

We can picture the scene: Attorneys and FBI agents, who have sworn to uphold the law attached to a machine that is said to be more truthful than they are. Men and women whose oral testimony or whose representations to a judge have sent ordinary citizens to prison being required to prove (by means of a machine) that they told the truth to an internal investigation.

That's bad enough, but there is more. Among this marvelous machine's potential subjects are some of those who at other times and in other places use the same kind of machine to test the truthfullness of suspected criminals. What kind of a world is it when those who are trusted to report truthfully on this machine's analysis of the answers of others cannot be trusted to report truthfully about themselves?

What happens if this machine -- which, its proponents claim, is almost but not quite infallible -- fails to spot the person or persons who leaked information? Interrogation under hypnosis? Sodium pentathol? From beginning to end the Abscam project seems to have had a special knack for bringing out the worst in everybody, not just those congressmen who were being videotaped. It is a hex.