Thirteen years ago, a book by French author J.J. Servan-Schreiber called "The American Challenge" started alarm bells ringing all over Western Europe. The book warned that unless Europe woke up, American technology, management and organizational skills would plunge the old continent forever into second-class status.

Today, Europe is wide awake and the challenge is to America. It comes from an increasingly independent Europe worried about the deadline of U.S. leadership and power, the growth of Soviet military strength and how to protect Europe's own security and impressive prosperity in a varstly changed world.

Among specialists closest to the situation, there is a view that relations between the United States and its major allies are the worst since the end of World War II.

It is a situation that neither side wants. Yet it continues to tumble along, seemingly beyond anybody's ability to control. It is fed by mishandling in Washington, lack of foritude in Europe, misunderstanding on both sides and legitimate differences of opinion on how best to handle a fistful of international crises.

Official spokesmen and some commentators point out, correctly, that the western alliance has had serious differences in its 31-year history about defense and nuclear policy, Vietnam, China and the Arab-Israeli wars.

But what is happening now is different, more serious, involving fundamental currents running through the alliance in different directions with no clear way to pull them together.

The differences are these:

In previous periods of disarray, U.S. policies may have been questioned but not U.S. power or its leadership role within the West.

Fairly or not, doubts about American leadership, competency and consistency are widely felt around the world. The doubts go beyond just the Carter White House to the question of whether Congress is immobilized, to the weakened U.S. economy and dollar and to a political process that makes Europeans, East and West, wonder if it still produces the best.

What may be unique and most important about these doubts is the fact that they are already several years old and could persist several more years, no matter who wins the November presidential election.

The United States has not had a full two-term president since the Eisenhower years. The last six years have been one president resign in disgrace, another serve without being elected and a Carter administration which, despite an enviable record in helping to beef up the NATO military alliance, has had a steady stream of disputes and real or perceived foulups with Europe, Japan and the other allies.

Many years, therefore, already have passed in terms of allowing new attitudes about the United States to take root around the world.

At the same time, the Soviet Union, by virtually all accounts, has grown even to or surpassed the United States in relative military power and has achieved the dubious political benefit of being seen as more willing to use that muscle to achieve influence.

Western Europe, meanwhile, has grown stronger and wealthier, led by the re-emergence of West Germany. Perhaps more importantly, however, there are signs that Western Europe is no longer exactly sure where it is going or what the Europe of the distant future will look like. It is a question that is being accelerated, maybe even caused, by the situation here.

West European materialism and democracy will no doubt keep the life-style and values tied to the western alliance and the United States for a long time. But the idea is afoot among some Europeans that on some political questions a more accommodating position toward the Soviets, one that puts them midway between Moscow and Washington, is at least worth thinking about. The idea basically seems to be that if the United States is in a prolonged decline, those who have prepared for it may be better off in the long run.

On top of this is another trend -- instability in less developed countries of the Third World.

Indeed, it is the continuing dual crisis caused by the taking of U.S. hostages in Iran and the Soviet invasion of Afghanistan that captures all of the fundamental new currents, focuses them on how the West responds, and leads to the grim conclusion that the alliance is facing its worst challenge.

Both crises, however, are outside NATO's formal geographic operating region in Central Europe and the North Atlantic. Thus the key question confronting the alliance on the 1980s will be whether and how the allies respond to Soviet military moves outside NATO's borders; in other words, outside of a direct threat to their own territory and well-being.

"I don't know how that question will be answered," said one U.S. official.

"But if Europe is perceived as only interested in protecting its affluence while the Soviets are on the march elsewhere, if they choose only to preserve the benefits of East-West detente through accommodation and refuse to take any bilateral measures, if that's the case, it will erode support [for the alliance] here and there will be a deep and dangerous trans-Atlantic split."

The current situation, adds another official, "is not just another crisis. It's a litmus test."

The Soviet invasion of Afghanistan last December was not only the first use of Russian troops outside of Communist East Europe in the post-war era. It also was precisely the kind of attack -- one on the flanks in an area where the West is weak rather than against the hard core of NATO strength in central Europe -- that specialists believe reflects a salami-slice kind of strategy Moscow is apt to use for the future. Thus, coping with it presents a severe test.

As an alliance, NATO is built upon collective security. In that role it has performed well in Europe. The politically tough decision taken by NATO countries last December to allow stationing of new U.S. atomic missiles on their soil demonstrated that resolve to protect Europe.

But oil has driven a wedge into the alliance. All of NATO's 15 member nations, plus Japan, have a huge stake in the stability of Southwest Asia and tne neighboring oil-rich Persian Gulf.

The Europeans, however, live in the shadow of the Russian bear and have different and more passive ideas about how to deal with Moscow over Afghanistan. They also have lucrative commercial trade with the East and concerns that a new Cold War could shut off human contact and return tensions to the old battlefields of Europe.

Thus, with the exception of verbal condemnation -- and the willingness of West Germany, Canada, Norway and Turkey among NATO members to join the Olympic boycott -- the allies have taken no substantive, punitive measures against Moscow.

Furthermore, the Europeans are likely to begin pushing their own formula for a Middle East peace soon, involving recognition of Palestinian self-determination, another move contrary to U.S. efforts but which would be received well in the oil states.

Finally, there is no forum for trying to straighten out this dangerous mess outside the alliance borders.

NATO, because of its formalized boundaries and the certain objections of some member states such as France, cannot serve as the place for developing Southwest Asia strategy.

European officials of the nine-member Common Market meet regularly every two weeks. Foreign ministers from those countries meet monthly and the heads of government four times a year. In part, these frequent meetings of Europe's heirarchy reinforce many of them in a common critical view of the Carter administration.

The United States, however, only gets a high-level collective crack at them during the semi-annual NATO meetings, which are inappropriate for discussing Southwest Asia, and at annual summit meetings. But these summits thus far are meant to discuss economic issues and are limited to the seven major western industrialized powers.

Forming Persian Gulf strategy in such a forum, which excludes the smaller European countries, could weaken support of those nations for sharing the alliance burden generally.

In Washington, some officials share the European view about Iran, namely that economic sanctions at this point won't get the hostges out and will drive a country, bound at some future time to be important to the United States, further away from the West.

By and large, however, there seems to be a wider feeling that many countries have disgraced themselves on the Iran of Afghanistan questions and that the world, and its values, looks topsy-turvy.

In both situations, fundamental principles -- the safety of embassies and diplomats and the invioability of borders -- have been violated.

Yet, just this past week, the following things happened:

In Pakistan, many of the foreign ministers from 40 Moslem countries meeting there tended to equate Soviet "aggression" in Afghanistan with U.S. "aggression" in Iran. This, even though Moscow invaded a Moslem country of 15 million people, killing thousands of them, while the United States tried to rescue illegally detained hostages, killing eight people, all Americans.

In Italy, European Common Market ministers watered down economic sanctions against Iran from those previously pledged and then the British Parliament backed away even from the reduced sanctions.

Olympic committees in Belgium, Holland, Italy, Austria, Ireland, Portugal and Sweden voted not to go along with the United States on the boycott of the Moscow Games. Committees in France and England, just a few days earlier, also had decided, in opposition to their governments, to send teams to Moscow this summer.

Of the major European powers, only West Germany stuck with the United States, a move that now could isolate Germany in Europe and cause some internal problems for the Bonn government.

For the moment, the administration has succeeded in pushing the Iranian situation out of the headlines. Later this summer, the Olympics will be over. The hostages in Iran may also eventually be released. So at some point, some of the current and lingering flashpoints will pass. But, specialists say, the long-term trends remain "clearly troublesome."