The United States has instigated private talks with some but not all of its major allies seeking a basic overhaul of international trading rules because the existing system is not coping with growing trade tensions worldwide, Reagan administration officials said yesterday.

The talks were described here as "low key" and "informal." Yet they have been carried out at the highest level by American trade officials--U.S. Trade Representative William E. Brock, the administration's top trade negotiator, and his two deputies, Michael B. Smith, formerly based in Geneva, and David Macdonald, who has resigned and whose place will be taken here by Smith.

The administration's move is directed at the 88-nation General Agreement on Tariffs and Trade (GATT), which was formed after World War II to regulate international trade.

U.S. officials said they are troubled by the "lack of political will" displayed during November's GATT ministerial meeting. That session failed to come to grips with major trade problems, especially barriers that are being erected against the export of services that brought more than $100 billion to the United States last year, administration officials said.

The administration also is deeply disturbed by what it views as GATT's inability to settle trade disputes among its members. This was vividly illustrated, officials contended, by vacillation of a GATT panel dealing with a U.S. complaint that European government subsidies on exports of wheat flour have hurt America's overseas sales. The subsidies permit European farmers to undersell U.S. producers.

This failure to issue a clear ruling drew a sharp attack from Brock early this month.

If successful, the talks would bring about the first major changes in GATT rules since the Tokyo Round of the last decade led to the lessening of tariff barriers.

Administration officials said their aim is to strengthen GATT and they estimate that they have "at best" three years to do so before the process becomes overwhelmed.

The administration's move, however, is expected to anger some of America's closest allies, especially members of the European Community currently engaged in potentially dangerous trade skirmishes with the United States over agricultural subsidies, and many of the less-developed nations who see it is another attempt by the industrialized world to maintain economic dominance.

Administration trade sources said some members of the European Economic Community have been cut out from the talks. While these officials refused to say who was excluded, European sources said France--which has irritated many other EC members with its exclusionary trade posture--is not involved while West Germany and Great Britain have been consulted.

Divisions are likely to emerge Thursday when EC ministers meet here with Brock in an attempt to settle the agricultural differences at a high political level before they escalate.

Europeans are privately warning U.S. trade officials that there will be serious reprisals if the United States makes another move such as the recent U.S. sale of $150 million in government-subsidized wheat to Egypt. That cut France out of what it had considered its traditional market.

But U.S. Agriculture Secretary John Block indicated yesterday that there will be other such moves if Europeans don't agree to restrict their sales of subsidized farm goods.

While these bilateral disputes engage much of the world's attention, administration officials feel the real issue is the way GATT operates.

Three broad areas of possible change have emerged, to prod the GATT nations into a serious consideration of the organization's future.

One, which bears the working name of "GATT-plus," would constitute a group of nations that would agree to engage in freer trade than called for under GATT itself. They could, for instance, decide to have completely free, two-way trade between themselves with no tariff or non-tariff barriers. The problem, however, is this could upset most favored nation treaties now in force.

Another, which has been called "super GATT," would forge a group of nations to exercise trade leadership, hoping to implant their views in the existing GATT system so that the rest of the trading world will join.

The third, called "the GATT of the like-minded," would see countries agreeing to cut barriers and inviting other nations to join in.

Among the nations reported to have been talked with by the U.S. are Canada, Japan, Finland, Norway, Sweden, Iceland, Brazil, South Korea and the ASEAN block of Indonesia, Singapore, Malaysia, Thailand and the Philippines.