IN DAY-TO-DAY criminal investigations, J. Edgar Hoover had a rule against the FBI immersing itself in the underworld to catch crooks, and for good reason: He worried about his agents being corrupted through association with hoods. For the same reason he resisted FBI responsibility for drug enforcement, which often means planting agents in drug rings.

In recent years, however, Hoover's successors have been going all out in the opposite direction. Very much "in" at the FBI today are stings, scams, infiltrations, the testing of public officials with bribe offers, and the use of "middlemen" to go after anyone who might succumb to invitations to commit a crime.

The FBI budget for undercover work has jumped from $1 million in 1977 to a requested $6 million in 1982. Undercover operations have climbed from 53 in 1977 to an average of 300 yearly between 1978 and 1981, a stunning increase that is applauded by the Reagan administration.

Law enforcement "must interject its agents into the midst of corrupt transactions," Attorney General William French Smith has declared. "It must feign the role of corrupt participant. It must 'go undercover.' "

Seemingly forgotten is what happened when Hoover broke his own rule and encouraged FBI agents to go undercover en masse to investigate "subversive activities." As a Senate committee investigation of those days found:

"The abusive techniques used by the FBI in COINTELPRO from 1956 to 1971 included violations of both federal and state statutes prohibiting mail fraud, wire fraud, incitement to violence, sending obscene material through the mail, and extortion.

"The government, operating primarily through secret informants, but also using other intrusive techniques such as wiretaps, microphone 'bugs,' surreptitious mail opening, and break-ins, has swept in vast amounts of information about the personal lives, views and associations of American citizens.

"Unsavory and vicious tactics have been employed -- including anonymous attempts to break up marriages, disrupt meetings, ostracize persons from their professions, and provoke target groups into rivalries that might result in deaths."

These and other disclosures resulted in elimination of most of the FBI's domestic intelligence program. Today, instead of hundreds of thousands of open FBI files on American citizens, there are perhaps 100 -- all connected with real criminal activity.

There is no doubt that undercover methods can be extraordinarily effective at times in snaring those engaged in both violent and economic crimes, from narcotics trafficking, labor racketeering and truck hijacking to arson, white-collar frauds and political corruption. But we badly need a careful and dispassionate examination of the immense spread of such operations by federal prosecutors and by the FBI -- indeed, by all U.S. police agencies -- and of the consequences.

One reads regularly of new problems. Innocent Americans are claiming that they have been severely damaged by scams and offshoots of scams. Hundreds of millions of dollars in lawsuits have been filed against the government. Corruption in the FBI has been charged. And this is to say little of the damage done to the very fabric of our society when undercover operations are not tightly controlled.

Consider how innocent people can be hurt.

In "Operation Frontload," the FBI employed a convicted criminal to investigate the construction industry. To obtain certification for him as an insurance agent with the power to issue bonds, the FBI told the insurance company he was as a former police officer and "a straight arrow." The insurance company wasn't told his real name, his criminal record, or that he had agreed to be an informer to avoid a nine-year prison sentence.

The result: The insurance company claims the FBI's man issued fake performance bonds and cost the insurance company and the insurance broker more than $60 million in business losses.

In fact, 12 lawsuits seeking damages of more than $340 million have been filed just as a result of "Operation Frontload." The government has already paid more than $800,000 in settlements.

Similarly, in Abscam, FBI middleman Joseph Meltzer used the scam framework of Abdul Enterprises, Inc. to work private swindles of his own in California and Colorado. Fearful of blowing his cover, the FBI did not stop Meltzer's private cons. Innocent businessmen and women were swindled, reputations were ruined, and two of the victims contemplated suicide.

Or consider how law enforcers themselves can be corrupted.

In Chicago, a policeman posed as a pimp and infiltrated a prostitution ring. After the case was over, he continued his pimping and was suspended.

In Abscam the FBI agents supervising Mel Weinberg -- the convicted swindler to whom the FBI paid more than $250,000 -- denied in affidavits that they had any business dealings with Weinberg. Later they admitted buying furniture, clothing, and books from Weinberg, and one agent admitted receiving an automobile from him. Marie Weinberg, Mel's wife, now deceased, also swore that her husband covered for the sexual escapades of agents.

That corruption of undercover agents happens from time to time, while unfortunate, is not surprising. The agents are put into criminal environments heavily infected with temptations and risks. They lack such usual controls as uniform, fixed place of work, and visible supervisor. They sometimes play crook, and each hour of the operation can make them more susceptible to the rationalization of illegal or immoral conduct. But the degree to which such corruption now seems to be occurring should be deeply troubling.

In short, everybody can lose in these operations -- not least the FBI. As Harvard Prof. James Q. Wilson has remarked, J. Edgar Hoover "knew that public confidence in FBI agents was the bureau's principal investigative resource and that confidence should not be jeopardized by having agents appear as anything other than well-groomed, 'young executive' individuals with an impeccable reputation for integrity."

The classic FBI agent is neatly dressed, clean-shaven, without paunch or scuffed shoes. He drives a four-door car that your teen-ager would borrow only in desperation. For more than half a century he, occasionally she, has been solving crimes with traditional skills developed and refined by the police of democratic countries -- skills in interrogation, in following leads, and increasingly in the use of science and laboratories.

But that is not the picture of the FBI that has emerged from the surge of undercover scams.

The Justice Department has attempted to grapple with these issues through guidelines issued in 1981 by former Attorney General Benjamin Civiletti. While certainly a step in the right direction, the guidelines raise some serious questions of their own.

One reads with alarm, for example, a paragraph permitting the FBI director to authorize "otherwise illegal activity involving a significant risk of violence or physical injury to individuals." The same paragraph also allows illegal actions by undercover agents when necessary "to establish and maintain credibility or cover with persons associated with the criminal activity under investigation."

These extraordinary provisions seem to authorize the FBI director to approve any type of criminal activity by undercover agents -- burglaries, assaults, even murders -- when necessary to maintain cover.

Even the strictest guidelines, moreover, can be weakened at the whim of an attorney general. The guideline provisions on FBI anti- subversive investigations, for example, are currently under assault by conservative columnists and politicians who view them as too restrictive. These critics would like the FBI to return to the days when hundreds of thousands of files were kept on "radicals" and "radical"organizations, with FBI-employed informants authorized to disrupt the groups.

To FBI Director William H. Webster's credit, he has so far resisted demands to weaken the guidelines, but what will happen in the future is far from certain.

We could learn a thing or two in this regard from some other nations with considerable success in combating crime while maintaining a free society.

Danish police, for example, use undercover agents, but only to gather information. In Denmark the offering of bribes by undercover police would be an unlawful or undesirable use of agent provocateurs. The Danish prosecutor cannot legally grant a person immunity from prosecution. Hence, an agent provocateur such as those used by the FBI in scams and stings would be punishable as an accessory to the planned or committed crime.

The rules in Sweden, Holland and Norway closely follow the Danish laws.

Nor are Japanese police permitted to go underground to fight crime. There the police, considered moral actors and models for the young, go to great lengths to be blameless citizens of the neighborhood, with members of the public considered assistants in reducing crime.

The fact is that it will be difficult, and perhaps impossible, ever to resolve the issue of what the FBI or police can or cannot do in undercover work in this country by relying solely on guidelines and the bits of existing law.

Fundamental constitutional issues are at stake, in particular the First and Fourth Amendments.

The essence of the First Amendment is that speech must not be interfered with by the government, that Americans must be free to communicate with one another without the fear that a secret government agent has infiltrated their homes, offices, schools, churches, or political gatherings.

The Fourth Amendment guarantees that Americans are secure in their homes, persons and papers against unreasonable government searches. Ordinarily, when the police or FBI want to search your home or office or tap your telephone, they must obtain a warrant from a judge based on a finding of probable criminal behavior.

It is hard to imagine a more intrusive device than an informant hired by the police or FBI who worms his or her way into the target's confidence, who follows the target from place to place, who becomes a trusted friend or lover, who even encourages criminal conduct. Yet precisely such practices are permitted under current law -- with no requirement that the target be suspected of criminal activity.

While this government-directed conduct is significantly more intrusive than a telephone tap, the Supreme Court has not found such use of informants or middlemen to violate Fourth Amendment guarantees against government intrusion.

Undoubtedly there are undercover activities that would not require a warrant. Currently the Fourth Amendment is not construed to require a warrant in every case; a federal officer may search without a warrant while conducting a valid arrest, for example, when it is necessary to prevent the destruction of evidence or to seize weapons that might be used to assault the officer.

But no laws have been enacted in the United States to establish procedures for undercover activities and the use of informants, middlemen and agents provocateur. And it is procedure that spells much of the difference between rule by law and rule by whim or caprice. Most provisions of the Bill of Rights are "procedural."

Procedures that would resolve some of the problems discussed here clearly should be enacted into law by Congress. It is, of course, unrealistic to think that this will happen soon; the political climate is not right. But the examination must be continued, the evidence accumulated, the principles finely honed so that they are ready when the time is right.

In the meantime, we can only hope that we will not see many more innocent people hurt,,more damage done to the reputation of the FBI and other law enforcement agencies, or more citizens afraid that government agents may be lurking in their homes, businesses and other corners or their private lives.