The United States, which had promised to participate "constructively" in a new round of global economic negotiations, has started off by playing the role of a die-hard defender of the old order and spoiler of initial efforts to get a dialogue under way between the world's rich and poor nations.

At the United Nations special economic session, which ended in New York earlier this week, the United States found itself isolated with Britain and West Germany in a wrangle over the ground rules for the negotiations.

Eventually, the three caused the meeting to end with no agreement on these issues, leaving in considerable doubt the fate of the global economic conference that is scheduled to begin in New York in January.

The battle over ground rules involves what is emerging as one of the fundamental issues facing the global negotiations, namely which institutions and nations have ultimate control over the process of reforming the world economic order.

The dispute caused the session to be extended four times in a futile effort to break the deadlock, and prevented any discussion of the agenda. The single the 1980s, which amounts to little more than a general declaration of aspiration and has no binding effect on any government.

Reports circulating at the United Nations said that U.S. Ambassador Donald McHenry sought to get the administration to accept a compromise Yugoslav-sponsored resolution but that he failed, despite several phone conversations with President Carter.

Observers noted that in two of the three holdout Western countries, the United States and West Germany, the incumbent leaders were facing elections and a challenge from conservative opponents who might well have capitalized on anything smacking of concessions to the Third World.

This was presumed to be a major factor in the hard-line American position, though administration officials insisted there was an extremely important issue at stake.

The initial U.S. strategy called for highlighting the special responsibility of the Organization of Petroleum Exporting Countries for the current world economic crisis because of spiraling oil prices. The United States had also hoped to enlist the support of the world's poor, oil-importing countries in pressing OPEC to provide relief, a move aimed at splitting the Group of 77, the body representing 119 developing countries on trade and aid matters.

However, no such split materialized. Furthermore, the focus of the session's attention quickly shifted from the energy issue to the contested procedures for holding the global talks, with the opposition of the three Western powers emerging as the principal roadblock to an agreement.

Behind the struggle over procedural rules lies the real bone of contention between the Third World and the industrialized Western nations: a radical reform of the existing international economic order.

The Group of 77 is trying to force through this reform by centralizing the economic global negotiations in a single body. Its authority would be greater than the already existing U.N. special agencies, notably the Washington-based International Monetary Fund (IMF) and World Bank, which now wield enormous influence.

Because of the weighted voting system used in these organizations, the United States and other Western nations have a dominant say in policy-making and the allocation of funds. They are not anxious to see the status quo changed.

The Yugoslav compromise sought to get around this dispute by empowering the central conference to create ad-hoc bodies to deal with the individual issues and allow even the specialized agencies to draw up proposed solutions.

But the wording of the Yugoslav draft resolution was "too vague" as one U.S. official put it, to overcome the opposition of the three Western nations.

The United States specifically objected to a statement that the central conference body would be a forum for "coordinating and conducting" the negotiations and another that said it would receive the reports of the specialized agencies and ad-hoc bodies "with a view to reaching a package agreement."

"The language was unacceptably vague," said one U.S. official familiar with administration thinking. "It just sounded too centralized for us."

"We believe the world is well served by having different agencies having different authorities and competencies," he added.

Various attempts to satisfy the United States on the point of safeguards for the existing agencies all failed. Canadian Ambassador Michael Dupuy said in a telephone interview that the Yugloslav draft had also tried to assuage American fears by including wording that "all important issues" would be adopted by consensus, meaning the United States could veto any proposal.

Dupuy said the draft in addition defined in "a tight, legal way" all the issues that would be considered "important" as a further guarantee.

Essentially, the United States is still extremely wary of the Third World's openly stated objective of storming the whole system of monetary, financial and economic institutions set up at the end of World War II and largely controlled since by the Western industrialized nations.

"The Third World wants to break these down and get control," said one U.S. official.

The real issue, said another, is "whether or not you preserve the integrity of the specialized agencies."

Administration officials all insisted that the United States was not alone in this concern and that other Western nations simply differed over tactics, feeling that it was better to fight later in the conference over the substance of the issue rather than now.

They insisted that the administration was still acting "constructively," as Secretary of State Edmund Muskie promised in his speech at the session's opening.

"It is not 'constructive' to do in the system of specialized agencies," said one. "If you don't call a spade a spade, you create false expectations."

Having done just that, the fate of the economic global negotiations remains to be decided by the regular session of the General Assembly, which began Tuesday. But it has a full plate ready, leaving uncertain how or when the deadlock will be broken and permit the talks to get under way.