The Carter administration, after a savage bureaucratic struggle, reversed its negotiating position just before an important round of talks with the Soviet Union earlier this month on controlling the sale of conventional weapons.

The internal struggle, the policy reversal and the fruitless round of discussions with the Soviets in Mexico City Dec. 5-15 have placed the future of the negotiations in doubt. And unless the United States can obtain a degree of cooperation from the Soviets and other major powers. American officials agree. President Carter's arms restraint policy sooner or later will have to be watered down or abandoned.

The story of what happened and why is a tangled one involving matters of high policy and diplomacy, bureaucratic competition and even personal pique.

The U. S. delegation to the Mexico City talks, in a rare if not unprecedented restriction, was ordered to walk out if the Soviets brought up the question of arms restraint in Korea or the Persian Gulf. This was the case even though it was at U.S. initiative that the Russians agreed in the first place to focus discussion on specific regions of the world rather than on generalities.

At one point late last week, passions became so high that presidential assistant Zbigniew Brzezinski suggested by cable that the chief U.S. negotiator, Leslie H. Gelb, should pull out of Mexico City and come home. Gelb fired back a defense of his position and the "suggestion and the "suggestion" was reversed.

Reducing the worldwide traffic in arms was proclaimed by President Carter to be among his highest foreign policy priorities upon assuming office, and his personal effort to check the tide of rising arms sales has been a distinguishing feature of his presidency.

His method of first impossible unilateral U.S. restraints and then seeking agreement from other nations has had little success so far. Late last month he announced for the first time that the future of his self-imposed ceiling on U.S. weapons sales abroad will depend on the extent of cooperation from other arms-supplying nations.

The united States (38 percent) and the Soviet Union (34 percent) were the two top suppliers of arms to the Third World during the 1970s, and thus for reasons of size as well as rivalry the Russians are crucial to the question of arms restraint worldwide. Moreover, European nations have taken the position that it would be senseless for them to curb their arms sales until the United States can obtain Soviet agreement to restraint.

In March 1977, Moscow agreed to establish a working group with Washington to discuss the matter. Later that year the two sides began bilateral negotiations on "conventional arms transfers,"known in the bureaucracy as the CAT talks. There have been four rounds of negotiations, the latest being the just-concluded sessions in Mexico City.

In the beginning the chances for any success were rated at about 100 to 1 by U.S. officials, and the original discussions of broad "principles" was fruitless. At the suggestion of Gelb, who is director of the State Department's Bureau of Politico-Military Affairs, the United States with Carter's personal aproval proposed that negotiations focus on specific regions where actual restraints could be observed by the superpowers, eventually joined by other supplies.

The Soviets intially refused to consider regional restrictions, but the United States insisted that getting down to real cases was the test of Soviet sincerity. Eventually the Russians reversed themselves and agreed to regional talks.

The United States proposed to start with Latin America, where American sales of heavy weapons have been limited and where the Soviets have few arms clients, and with Africa, where the Russians have been pouring arms into Ethiopia and Angola.

The Soviets, on the other hand, wanted to talk about areas on or near their borders-Itan, their southern neighbor where the United States has supplied billions in the latest arms; China, their arch-rival which is bidding for western arms; divided Korea, where a U. S. buildup in the south threatens to generate a new supply of sophisticated Soviet weapons to the north.

At first few U.S. officials took the CAT talks seriously, but as the Mexico City sessions approched there seemed a chance for the first time for serious negotiations covering several areas of the world.

This was an exciting prospect for a few people committed to the talks and the global policy of arms restraint, but an upsetting prospect to many more: Brzezinski and other officials leery of collaboration with the Soviets; regional bureaus in the State Department and other agencies seeking to protect the interests and sensitivities of their clients; officials in the Pentagon and elswhere who feel international arms curbs are naive and impossible.

The issue came to a head Nov.27 at a White House meeting of the Security Coordination Council, a top-level unit of the National Security Council. Brzezinski reportedly opened the session with a strong attack on the CAT talks. Defense Secretary Harold Brown also attacked discussions with the Soviets on sensitive regions.

Gelb fought back unsuccessfully. His position was weakened by the absence of Secretary of State Cyrus R. Vance, who had abruptly canceled his planned schedule for the duty and flown to New York in a last-ditch attempt to sway South Africa on the disputed elections in Namibia.

After further policy wrangling during which Brzezinski reportedly took the controversy to Carter, the U.S. delegation to the CAT talks was ordered not even to listen to a Soviet discussion of arms curbs in the Persian Gulf, China or Korea. If the Soviets insisted , the U.S. delegation was instructed to walk out.

In Mexico City, the Soviet delegation headed by Ambassador Lev Mendelevich was incredulous, the angry at the sudden U.S. switch in signals and refusal to even hear what the Soviets had come to Mexico to say.

Gelb sent several cables to Washington reporting on the threatened breakdown in the talks and suggesting ways to salvage the negotiations. These were taken by officials in Washington as appeals from the original hard-fought decision and, in effect, rejected.

Late last week the Soviets proposed to discuss a related issue, the Indian Ocean. Negotiations on this subject had been broken off by the Washington because of the Soviet buildup in Africa. Gelb cabled for instructions, but was told to do nothing. He responded heatedly.

Around midday Thursday, Brzezinski dispatched a cable abruptly telling Gelb that he seemed to be having difficulty carrying out presidential policy, an therefore should consider coming home so the NSC official could "explain" it to him. Gelb fired back a chapter-an-verse recitation of his actions in conformity with stated policy, but said he would immediately come home. Within hours, a new cable was dispatched telling Gelb to stay on.

Vance, who was in the Middle East at the time, said by telephone yesterday that "I've checked into this since coming home and am totally satisfied that Gelb did not exceed or depart from his instructions." Brezezinski declined to comment.

The Mexico City talks ended last Friday with little success and little prospect of progress in future rounds of negotiation. Several officials on various sides of the dispute said the Carter administration must now reconsider what it wants to obtain from the conventional arms talks with the Soviets, and how it proposes to go about it.