The countries of the North Atlantic Treaty Organization failed to settle today on a common response to the Soviet destruction of a South Korean airliner.

France, Greece, Spain and Turkey refused to endorse the only concrete sanction proposed by the United States and other allies--a two-week ban on any civilian traffic to and from the Soviet Union, according to NATO sources.

But at least 12 of NATO's 16 nations decided to endorse the sanctions, West German Foreign Minister Hans-Dietrich Genscher said in Madrid.

Japan today announced modest sanctions against the Soviet national airline, Aeroflot, including a ban on landing of charter flights in Japan.

Some of the countries that refused to endorse the sanction, such as Turkey, cited bilateral agreements with the Soviet Union that prevented their participation in a joint retaliation against that superpower, the sources said.

Turkey said it was tied by the Montreux convention, sources said. French diplomats said earlier in Paris that their country was bound by the terms of a civil aviation agreement with the Soviet government and could not unilaterally stop Aeroflot flights.

Following the two-hour discussion at NATO headquarters, Joseph Luns, secretary general of NATO, said the 16 allied countries would announce separately their protests of the airliner disaster.

Luns said NATO had decided against a joint action against the Soviet Union because the problem could not be reduced to a conflict between the two superpowers.

Because pilots groups in various NATO countries had already decided to cancel flights to the Soviet Union for 60 days, a NATO embargo might be moot in some cases, Luns said. He did not spell out the individual sanctions that NATO countries were planning to take.

A U.S. State Department official said the condemnation of the Soviet Union should be as broad as possible and include countries, in Asia for instance, that are not members of the alliance.

He said all NATO governments had supported discussions at an emergency meeting of the International Civil Aviation Organization in Montreal next Thursday. The organization wants to define new air traffic conventions that will rule out the use of military force against civilian airliners in peacetime. NATO countries are expected to insist that the Soviet Union, which is a member, agree to the new language.

The NATO countries had also endorsed Korean demands that the Soviet Union provide a formal accounting of what happened, monetary compensation to Korean Air Lines and the families of the 269 victims, an apology for its action and "credible guarantees" from the Soviet Union against future repetitions, the official said.

Washington, he said, viewed European gestures, however limited, as a clear political "signal" to Moscow. Sources said, however, that Britain felt the NATO response was inadequate.

From Tokyo, Washington Post correspondent William Chapman reported:

The modest anti-Soviet sanctions imposed today by Japan in retaliation for the plane tragedy reflect a determined government effort to contain the affair within limits that will not damage overall relations with Moscow.

A guiding principle is that Japan should neither get out front of Western countries in retaliating nor fall too far behind. It is a familiar posture, reminiscent of the slow-moving Japanese response to the Soviet invasion of Afghanistan.

The sanctions announced this morning ban chartered flights to Japan by Aeroflot and prevent public officials from using regular Aeroflot flights. The government also appealed for a public boycott of regular Aeroflot flights.

There was no official American appraisal in Tokyo of these measures, but there were indications that the United States had anticipated something with more teeth in it.