Out of the Senate's special investigation of American intelligence operations in 1975 came the phrase "plausible deniability"--that is to say, no U.S. fingerprints on covert activity. Whatever one thinks of the old, Cold War way of countering Soviet incitement to insurrection and other dirty tricks with comparable U.S. countermeasures, success rested on secrecy.

And that is why--leaving aside its wisdom-- the Reagan administration's policy in Nicaragua is procedurally implausible, if not fatally flawed. Nowhere is this better illustrated than in a recent editorial in The New York Times which poses the question: "Why won't the Reagan administration debate openly and seriously the legality of an American-sponsored secret war in Nicaragua?"

The simple answer is that "secret" wars, by definition, are secret; they are not "declared." That being so, their legality can hardly be openly debated. Yet that's what we are getting: a hot and heavy debate on the legality of the U.S. role in a "secret" war that you can read all about in your newspapers and watch on television--a U.S.-supported guerrilla war against the leftist Sandinista government in Managua.

The Reagan administration does not flat-out deny its involvement with training, arms and other supplies. The president insists this is for the purpose of interdicting Nicaraguan support for the rebels in El Salvador and therefore does not violate a law passed by last year's lame-duck Congress barring U.S. activities "for the purpose of overthrowing the government of Nicaragua. . . ." But since the United States is helping former national guardsmen of the ousted Somoza dictatorship who are openly committed to overthrowing their own overthrowers, that's a transparent dodge.

As for the administration's unofficial explanation that it is only trying to give the Nicaraguan government a taste of its own medicine by way of discouraging Nicaraguan interference in El Salvador, that simply makes the policy foolhardy. A clumsy, inconclusive U.S. hand in the anti-Sandinista uprising compromises whatever reasonably moderate opposition forces there may be working in restraint of leftist extremism in Managua. It provides, as well, precisely the sort of "external threat" from the Yanqui colossus that almost always serves to strengthen repressive Latin regimes.

At best, this tit-for-tat could conceivably provide bargaining leverage in some larger multinational peacemaking effort in Central America--if that is truly the administration's purpose. A less noble objective, strongly suspected by congressional critics, is that the administration wants to establish a credible opposition movement in Nicaragua and then argue for a negotiated peace. The theory is that U.S. liberals would be horrified at the thought of doing business with the old Somoza crowd--undercutting their case for a comparable "dialogue" with the rebels in El Salvador.

Whatever the objectives of the administration's effort in Nicaragua, the developing mood in Congress is such that it may well self-destruct by its own heavy hand. To see why requires a little recent history.

One of the principal conclusions of the investigation conducted by the so-called Church committee eight years ago was that the CIA had broken free of effective congressional oversight. "Select" committees on intelligence were accordingly established to play the watchdog role. Working within this system, Chairman Edward P. Boland of the House Intelligence Committee last spring secretly attached a classified addendum to an appropriations bill. It would have restrained the United States from efforts aimed at ousting the Sandinista government.

That done, Chairman Michael D. Barnes of the House Foreign Affairs subcommittee on Latin America was vigorously pressured by Speaker Tip O'Neill not to push a stronger amendment publicly. It would have denied U.S. funds for direct or indirect support of any "military or paramilitary operations in or against Nicaragua." Instead, everything was to be done quietly, behind closed doors, in keeping with the new and presumably safer oversight system.

But then last November the administration's covert designs burst into public print: Newsweek laid bare compelling details of the "secret war." So did other publications. What had relied for its effectiveness on being covert had become titillatingly overt. So Boland went overt as well, following pretty much the language of his "classified" amendment. Thus the provision passed by the House late last year without a dissenting vote, and enacted into law.

And thus the outcry in Congress right now. Sen. Daniel Patrick Moynihan, the vice chairman of the Senate Intelligence Committee, says he finds a number of his colleagues questioning whether the CIA is breaking the law. Rep. Wyche Fowler Jr., who is chairman of the House Intelligence oversight subcommittee and has just returned from a Central American tour, believes that "under the best of circumstances" the Boland amendment "is not being fully adhered to." Committee investigations are under way or planned in both houses of Congress.

Meanwhile, the House Intelligence subcommittee last week approved the tougher version of the Boland amendment that Barnes was talked out of pushing last year. Finally, there is beginning to be increasing consideration in Congress of a whole new look at the oversight procedure. What we are seeing is a breakdown, in plain view, of the delicate and discreet arrangements worked out post- Vietnam and post-Watergate for a restraining congressional hand on the executive branch's supposedly secret operations overseas.