The United States agreed yesterday to conduct a "detailed analysis" of the effect upon the European Economic Community of U.S. restrictions on imports of specialty steel.

In response to European demands for compensation for losses because of the curbs, American trade officials acknowledged the United States may owe some compensation to the EEC under the General Agreement on Tariffs and Trade (GATT).

President Reagan imposed tariffs and quotas on all specialty steel imports to the United States on July 5 in an effort to protect the industry from "the pervasive nature of unfair trading practices around the world."

Officials of European governments immediately attacked Reagan's action as protectionism and said their specialty steel industry was being used as a scapegoat.

The Europeans demanded $600 million in compensation over the next four years, contending the cost to them of the U.S.-imposed import restrictions could exceed the amount of the Europeans' unfair trade practices by that much.

An American trade official branded the European claim as "not realistic. The Europeans asked for the compensation in the form of reduced tariffs on their shipments of steel and steel-related products, chemicals and textiles to the United States. The United States already assesses countervailing duties on European specialty steel to make up for their governments' subsides.

"We may owe them something, though we don't admit to anything. But we are not going to pay for unfair trade practices," the American trade official said.

Separating the cost of restrictions that cover unfair trade practices from losses that are not justified under GATT's rules covering international trade, however, is a very complicated job that will likely take months. Yesterday's talks in Geneva were the second in a series with the third scheduled for next month.

When he imposed the import restrictions on specialty steel, Reagan said exporting nations could negotiate separate orderly marketing agreements with the United States with quotas and tariffs based on the level of unfair trading practices. The United States has started negotiations for orderly marketing agreements with Sweden, Austria and Canada; Japan has indicated an interest in also beginning talks. Trade officials said this option remains available to the EEC, though it chose to seek compensation.

If the EEC and the United States fail to reach an agreement, the Europeans have the right to retaliate, probably in the form of restrictions on U.S. farm sales in Europe.