U.S. Trade Representative William E. Brock said yesterday the ability of the General Agreement on Tariffs and Trade (GATT) to settle trade disputes suffered a setback when a panel vascillated on a U.S. complaint that European nations unfairly subsidize wheat flour for export.

"This failure to make a decision casts great doubt on the effectiveness of the GATT dispute-settlement process and undermines the limited discipline that now exists for agricultural trade," Brock said in his strongest public criticism of GATT.

Until now, the chief American trade negotiator had strongly defended GATT procedures against an onslaught of complaints from a protectionist-minded Congress as a key factor in preserving free trade in the world.

Brock's comments were sparked by a GATT panel report that he said backed the U.S. position that the European Community, usually known as the Common Market, gained an unequal share of the world wheat-flour market because of export subsidies. But the panel failed to take the next step of ruling that the subsidies violated the European Community's international obligations under GATT, Brock said.

"Thus it left the major legal issue unresolved."

Instead, the GATT panel urged in a nonbinding recommendation to the subsidies code committee that the Europeans limit their use of subsidies on wheat flour for export. Brock said that GATT committee "soon must answer the legal question left unresolved by the panel."

Julian Heron, an attorney specializing in agricultural trade issues who represents the Miller's National Federation that brought the trade complaint in 1975, refused to call the GATT panel decision a loss for American suppliers.

At worst if it is left standing, he said, it effectively mutes European complaints that the United States had violated international trade laws by its sale in January of $150 million worth of subsidized wheat flour to Egypt. The sale effectively wiped European suppliers, especially France, out of a market they had considered theirs and, according to Brock, was designed to pressure the community into ending its subsidized export sales.

Although the United States said the sale violated no international trade agreements, the European Community was reported Tuesday by Dow Jones News Service to have demanded $30 million compensation for the business it lost.