FOR OVER 30 YEARS the Western alliance has been obsessed with combating an unlikely threat with an incredible defense.

At no time since the establishment of NATO have military or intelligence experts rated the prospect of a Soviet military invasion as anything but remote. Originally, the military buildup in Europe was designed to be a "modest shield," as George Kennan put it, behind which recovery could take place. At the hearings on the North Atlantic Treaty held in the spring of 1949 John Foster Dulles, then serving as senator from New York, summed up the prevailing view, "I do not know of any responsible high official, military or civilian . . . in this government or any other governement who believes that the Soviet now plans conquest by open military aggression."

Today Soviet capabilities to mount an invasion of Europe are no doubt greater than in the past, as former secretary of defense Harold Brown stated in his final posture statement. But, he added, "I doubt that we would have to contend with such an assault as a bolt out of the blue." Even strong advocates of a buildup of U.S. military forces, such as Paul Nitze, rate a Soviet invasion as "unlikely" but worry that the Kremlin could use the theoretical capability of launching such an attack as blackmail. John M. Collins, a self-styled "conservative critic of defense" at the Library of Congress, says, "I believe the likelihood that the Soviet Union will attack Western Europe . . . in the short-term or even midterm future is extremely remote."

So for almost two generations the NATO alliance has been beset by schizophrenia. The actual military threat against which the allies continue to arm is rated low; yet military planning conventionally addresses capabilities, not intentions, which after all can change.

The military problem -- the theoretical possibility that the Soviet Union with its large standing army could launch hundreds of divisions across the Elbe -- has no satisfactory military solution, but the whole NATO panoply, including the U.S. divisions, the U.S. "guarantee" and the endless public discussions such as Secretary of State Alexander Haig's disclosure of NATO's "nuclear warning shot" strategy, are all designed as an elaborate insurance policy.

The trouble, however, is that the premiums have become much too costly -- not only because the monthly payments threaten the economies of the members, but because the payoff looks increasingly dubious. Nuclear "defense" is a sham, since the use of the weapons, with or without neutron warheads, will destroy that which is to be defended, whether or not the United States and the Soviet Union hurl missiles at one another.

A conventional defense is certainly more rational than a suicidal nuclear defense, but the most a conventional defense can do, assuming the Soviets make an all-out attack, is to delay their progress and to establish the political environment for a settlement. Given the nature of modern weapons, a short conventional war -- 10 days, six weeks or whatever -- would also devastate much of Europe. Why the combatants would be willing then to display the reasonableness thatt seems to elude them now in a time of peace is not clear.

For the Europeans and the Americans to have a big enough conventional force to raise the "nuclear threshold," the industrial states would have to be prepared for a sacrifice of social spending beyond anything they have yet instituted and to introduce permanent conscription on a massive scale.

While Ronald Reagan is pleased to proceed with the former, he is reluctant to institute the latter -- for the very good reason that in the United States conscription amounts to a popular referendum on foreign policy. In Europe, cutting government benefits such as social security and health care benefits that have been offered since Bismarck's time is politically dangerous even for conservative governments. Europe's only sure defense is peace, and that is why there is a huge peace movement on the continent.

The fundamental contradictions in NATO strategy explain why for more that 20 years so many weapons issues -- the Skybolt missile, the Multilateral Force, neutron bombs, cruise missiles -- have been highly divisive.

There are no good security answers for Rurope within the present conceptual framework. The allies have used weapons issues as metaphorrs, and in the process much of the discussion has the appearance of a charade. When Germans like the conservative Christian Democratic leader Franz Josef Strauss say to the Americans that the alliance must not weaken in its resolve to use nuclear weapons in Germany, it is not that they wish to turn their country into a radioactive wasteland. All they mean is that the fiercer the threat, the better the deterrent.

Whether that was ever a prudent defense policy is questionable, but it is manifestly imprudent now that the Soviet Union has the capacity to match whatever the West does with nuclear weapons. When the risk of war by miscalculation was less than it is now, irresponsible statements about nuclear weapons were less dangerous. The rise of antinuclear sentiment in Europe reflects a growing sense that with respect to defense strategy, the emperor has no clothes.

There are some fundamental problems with existing theories of strength through deterrence. The deterrence system rests on assumptions about human nature which are probably wrong: for example, that leaders with a finger on the button will act rationally under conditions of ever-increasing threat. Because of the creep of technology, a deterrence system based on an elusive "balance" is inherently unstable; every weapons laboratory development, real or imagined, has a potentially destabilizing effect on the relationship.

A security system is likely to be more stable if it is based on positive incentives to keep the peace rather than on threats, particularly since the threats are becoming less credible because of their suicidal implications. But we will be living under a security system based largely on deterrence for a long time to come, if only because the leading nations are so heavily invested in it and the smaller nations have aped the great powers in designing their own security arrangements.

But it is essential to build a deterrence system on strength, not weakness. A strategy based on nuclear weapons in which the public quite properly has little confidence divides the society and invites outside pressure.

Nor does a society become strong by sacrificing growth, innovation and the soundness of its economy to an endless arms race.

The failure of the United States to manage its economy has shaken confidence in American leadership around the world. if the richest society in the world cannot manage its own affairs, how can it defend others?

The Soviets pose a potential military threat which seems increasingly implausible. If the Kremlin cannot successfully control Eastern Europe with military force or pacify a wretchedly poor country like Afghanistan, why would they think that they could successfully occupy Western Europe?

But high interest rates in the United States, to take one important example, are threatening European economies now. Part of the premium we and the Europeans are paying for the military security system are the adverse economic consequences that affect both of us.

If there should develop a genuine fear of a Soviet invasion in Europe, then the only workable deterrent would be to create the impression in the minds of Soviet leaders that the invading or occupying troops face the same prospect which has kept them out of Yugoslavia and, so far at least, Poland: the prospect of permanent popular resistance. The stronger the society in real terms -- in its ability to manage its economy, to build popular solidarity, to arrange a secure resource base -- the more credible such a prospect becomes and the less inviting a target it makes.

Much of the so-called "pacifism" in Europe is based on a growing conviction that piling up military hardware that cannot be used without destroying the user does not add to national strength.

In the United States we have fallen into the trap of measuring increments of strength in dollars even though we have yet to have a serious debate inside the government or in public about the actual uses to which the new military hardware and military bureaucracies would be put. By diverting scarce resources to the military from investment in the productive system and social infrastructure, successive U.S. administrations have seriously weakened the American economy, which is the foundation of national strength, and damaged its educational system and resource base.

When Americans demand the Europeans increase their military expenditures by some arbitrary percentage, typically the call to sacrifice is defended not as a means of delivering a radically different military reality, but as one that will overcome the competitive advantage Europe and Japan have had by virtue of their relatively low military expenditures over the past two decades.

A much greater danger to the West than the unlikely Soviet invasion is an internal political, economic and moral crisis. Such weakness could indeed play into the hands of the Soviet Union.

If in a time of slow growth the European democracies are unable to deal with the historic structural problems they face, they run the risk of the same sort of social dissolution that caused the fall of France in 1940, when it boasted the largest army in Europe.

A dynamic Europe that is modernizing its institutions, managing its economy and establishing mutually advantageous relations with resource-producing countries, has little to fear from Soviet intimidation. In the test of nerves against which we are supposedly arming -- Kremlin blackmail -- economic and political strength are far more important weapons than more hardware.

If a society is not prepared to surrender, then it cannot be intimidated. If it looks strong internally and demonstrates high morale and vitality, it is less likely to be tested.

The whole "blackmail" theory needs a hard look. It is not clear why the Europeans -- who have been living 10 minutes away from Soviet city-destroying weapons for 20 years -- should suddenly cave because the Soviets have new weapons capable of making the rubble bounce. It is not clear what the Soviets could blackmail Europe into doing.

Leaving NATO? It is more credible that some European governments will leave NATO because we have scared them away than because of a stiff note or nasty phone call from Moscow.

Keeping Solzenitsyn off the television? That's what one distinguished exponent of the Finlandization thesis once argued to me. Is the fate of Europe going to hang on a question like that? If we have fools or cowards as leaders, no number of missiles will make the difference!

Reducing the industrial states of Europe to economic vassals? West Europe's proposed increased dependence on the Soviet Union for natural gas constitutes far more leverage than does the SS-20s.

Unless the Soviets are prepared to invade Europe -- and reduce to cinders the very assets they would hope to take -- then the blackmail threat is not credible, provided the internal economic and social strength of the continent is maintained.

The greatest threat to the capacity of the West to solve its security dilmena is the cripping psychological mood of the moment, based on the growing apprehension that nuclear war is inevitable. In the United States a recent Gallup poll showed that 47 percent of the population believes that a nuclear war within 10 years is either "fairly likely" or "very likely" and the overwhelming majority believes that their chances of personal survival in such event are no better than 50-50. In Europe the apprehension of nuclear war is even greater.

A security system in which the people to be protected have so little faith represents a serious vulnerability. The gathering pessimism is a consequence of the breakdown of negotiations and the attempt by the Reagan administration to make the deterrent credible by dramatizing the prospect of actually fighting a limited nuclear war.

Because of the inherent insolubility of the military security problem, we now face increasing danger of a serious split between the United States and Europe and a breakup of the alliance at exactly the moment it is the most needed.

The security options available to Europe in such a situation are both unsatisfactory. In the wake of a collapse of NATO, any accommodation with the Soviet Union would be on distinctly less advantageous terms than could be negotiated as a U.S. ally. A decision to build up European nuclear forces would neither end the arms race nor the growing popular revulsion against nuclear weapons which every Western government must now confront. All it would accomplish would be to insure an even greater Soviet nuclear threat against Europe under circumstances in which the Soviets might well be tempted to exert pressure on European governments.

The outlook and strategy of the Soviet Union can be tested by serious negotitions. It is now clear from eight years of SALT negotiations that agreements must be simpler, more comprehensive, and more quickly negotiated than in the past. A licensed arms race under which some systems are controlled while others leap ahead is not a practical policy.

It would be most welcome if the Kremlin were to accept President Reagan's offer to trade a promise not to deploy the Pershing II and cruise missiles in Europe for the dismantling of all the Soviet intermediate-range missiles in Europe. Everyone on both sides would be safer if such a deal could be struck. But shrimps will whistle, as Khrushchev used to say, when the tough bargainers in the Kremlin can be induced to trade over 600 missiles in place for a promise not to deploy weapons that, because of political opposition, may well be undeployable. The proposal was aimed more at European public opinion than at the Soviets, but it is unlikely to deflate the peace movement.

If the president wants significant reductions of nuclear weapons of all ranges, then a good starting point would be a freeze on the testing and deployment of all nuclear weapons delivery systems, which would include both strategic and European-theater nuclear weapons. The freeze, which could be for a specified period, say four years, could be verified by national intelligence means. The conventional argument that a freeze would preserve a Soviet advantage in Europe is not persuasive for two reasons.

First, there never has been a "balance" of land-based, middle-range weapons, and the U.S. abandoned the efforts to create one 15 years ago when we removed the last of the old-style intermediate-range rockets. If the NATO-assigned Polaris submarines are included, not to mention the French and British nuclear forces, there are enough European-based nuclear weapons to create the near-certainty in the minds of the Soviet leaders than an attack on Europe would result in the devastation of their homeland. The only "balance" of any significance is whether the capacity for retaliation exists on both sides.

Second, a four-year freeze would benefit the West. After all, it is the Soviets who are putting in a new missile every few days, not the United States. Indeed, it now appears that for technical reasons it may be almost three years before the first U.S. missiles arrive. It may well be that for political reasons they will never arrive. Without a freeze, the Soviets will be further ahead than ever.

Sooner or later, the United States, if we are really interested in arms reduction, will have to demonstrate our commitment to Europe by trading some of the cards we hold, such as forward bases, for the elimination of the Soviet nuclear threat against Europe.

Now that, as a result of popular pressure, President Reagan has at least opened up the possibility of a "zero option," similar pressure should be put on Moscow to make some unilateral reduction in the SS-20s. Such a gesture could make more likely the moratorium on deployment which is indispensable to a real reversal of the arms race. But the United States will have to come forward with less one-sided proposals than the Reagan offer. Were we prepared to act boldly, the alliance might once again serve to unite Europeans and Americans rather than to divide them.