When the Reagan administration's high command met last Thursday in an atmosphere heavy with gloom, it set criteria for possible withdrawal of Judge Robert H. Bork's nomination that could deepen the self-constructed fiasco.

Attorney General Edwin Meese urged that the nomination not be pulled. Chief of Staff Howard Baker responded that it would not, save under two circumstances. First, if Bork himself requested withdrawal; second, if there is "substantial" hemorrhaging among Republican senators, removing political benefit from a roll-call vote.

But how much is "substantial"? The three Republicans now listed against confirmation? A half-dozen? A dozen? The invitation to compound the Bork fiasco is clear. If a few more GOP dissidents are rounded up, nobody will have to go on record against one of the best qualified appointments ever made to the Supreme Court.

The problem lies with a White House mindset that has become dominant since Baker and his team took over last winter: defeat is a worse fate than retreat. Whereas the departed Patrick J. Buchanan viewed the Democratic takeover of the Senate as an opportunity for confrontation, Baker saw it as a mandate for conciliation.

The last chance for a political payoff from the Bork nomination may be lost. Not only has a new definition of mainstream been forged, but Reagan has been politically emasculated for his final months. What's more, southern Democrats may now satisfy the labor-black-schoolteacher coalition by opposing Bork without going on record in a roll-call vote. Nobody says anymore the Republicans can win this fight by losing.

How did presidential aides manage to grasp defeat from the jaws of victory on an appointment thought secure short weeks ago? They protest that the rules of engagement were changed. Instead of a set-piece confirmation campaign with the nominee making courtesy calls to senators, they were faced with a media assault of unprecedented ferocity mounted by the left.

But precisely this scenario was predicted to us in early July by worried officials at both the White House and Justice Department who considered Baker's strategy a formula for disaster. Their pleas for a mass campaign -- including a presidential speech to the nation -- got nowhere.

Attempts to mobilize the right were hampered by White House phobia about abrasive issues. It sloughed off a proposal that an anti-Bork petition by liberal lawyers be met by a petition not from conservative lawyers but from victims of crime. The left's ideological assault on Bork was not countered.

Even when it became clear the battle was being lost, its ideological core was not perceived. On Sept. 24, when Bork met with the administration high command to criticize its strategy, White House Communications Director Tom Griscom suggested a Barbara Walters interview of the judge and his wife to display his personal qualities. Bork and others said no thanks.

At the 11th hour, Bork's managers still played the old-fashioned confirmation game. Bork paid yet another visit to liberal Republican Sen. Arlen Specter, whose support had been made a special project by old friend Howard Baker. When Specter told Baker in a late afternoon chat Sept. 13 that he was still uncommitted, the president's men were cheered. Yet, GOP politicians back in Pennsylvania assured us a month ago that Specter could not and would not break with the civil rights lobby.

The lobby's triumph was signaled by published reports that a conservative southerner, Appeals Court Judge Patrick Higgenbotham of Dallas, leads second-choice possibilities. So, moderate Sen. Lloyd Bentsen of Texas, in announcing opposition to Bork Friday, suggested Higgenbotham instead. Actually, both Justice and the White House refuse to discuss alternatives to Bork, and Baker warned against such talk.

Still, the White House has not told fence-sitting senators they will get no fellow southerner if Bork falls. So, a typical southern Democratic senator, threatened with black and union primary opposition if he votes for Bork, feels no countervailing pressure. If many more Republicans defect, this hypothetical senator might be spared a recorded vote. And he could tell constituents he opposed Bork only to keep one southern seat on the court.