In what may be an ominous hint of a European turn toward trade protection, leaders of the nine Common Market nations today linked their unemployment problems to their "open and liberal commercial policy."

The nine, ending a two-day summit here, insisted that they "remain strongly attached" to the relatively free trade they pursue.

They instructed the Community's Commission in Brussels, however, to recommend measures to deal with five "sensitive" industries.

British Prime Minister James Callaghan, the summit's host, identified the "sensitive" sectors as footwear, textiles, ship building, steel and electronics. European firms in these industries have complained loudly of competition from Japan.

In effect, today's move lays the groundwork for protective barriers, such as quotas or tariffs, against imports. It is another sign that the prolonged high levels of unemployment in the West are straining the world's more-or-less free trade rules.

The Common Market action threatens to extend a system - started by the United States and emulated by others - including Britain of working out "voluntary" export quotas with Japan, the main target of Western unhappiness, over trade imbalance.

Many economists believe that the Great Depression of the 1930s was deepened and widened by the huge trade barriers thrown up by the United States, Britain, Germany and others.Each nation tried to protect its home market and domestic jobs by keeping out competing goods.

So after World War II, the industrial West, led by the United States, embarked on round after round of negotiations to bring down the barriers. Today tariffs and import taxes are so low everywhere that they no longer serve as major obstacles to trade, except in a few industries such as chemicals.

Until about 1970 world trade and prosperity grew at a remarkable rate. But the last seven years have brought a decisive change.

Every industrial nation is struggling with unusually high jobless levels that cannot be brought down without risking more inflation.It is against this background that the protectionist spirit is being revived on both sides of the Atlantic.

The five industries cited by Callaghan are particularly vulnerable to Japanese imports. Tokyo has aroused great irritation throughout the Common Market, where unemployment is now put at 5.2 million.

The Japanese are accused of unfairly pushing their sales abroad by selling goods at prices lower than they charge at home. At the same time Tokyo is charged by traders with maintaining an elaborate network of hidden, nontariff barriers to keep out foreign goods.

French President Valery Giscard d'Estaing reportedly wanted to adopt an even more strongly protectionist tone. But he was argued out of this, presumably by Chancellor Helmut Schmidt of West Germany.

The French did succeed in getting into the Community statement their favorite phrase these days - "structural changes in the economy." In Paris those words refer to any industry suffering a drop in demand and seeking special protection or support.

The trade statement by the nine, who also include Italy, the Netherlands, Belgium, Luxemburg, Ireland and Denmark, said:

"The European Council (the nine government chiefs) expressed its concern at the employment situation in certain sectors adversely affected by structural changes in the economy. It invited the Commission to continue studying all factors whether structural or otherwise - and to indicate the conclusions which it draws from the studies.

"The European Council also concerned itself with the effects on the employment situation of the open and liberal commercial policy of the Community to which the Community, as the world's largest importer and exporter, remains strongly attached."

Callaghan first underlined for reporters the group's adherence to liberal trade. Then he added, "But there are sensitive areas in our economies where a strict application of an open and liberal commercial policy can bring results that we really had not anticipated."

This was taken to mean that liberal trade is virtuous, but that a little protectionist sin may be necessary.

Yesterday, for the first time, the leaders called for the creation of a Palestinian "homeland." That declaration, worked out with Washington, puts all the leading Western nations in the same camp on this crucial issue, and it further increases pressure on Israel.