President Reagan's insistence that a ban on American-licensed equipment for the Soviet-European natural gas pipeline is "a matter of principle" poses a formidable task for trans-Atlantic diplomatic ingenuity, and for the vaunted mediating abilities of his newly chosen secretary of state.

The outcries from the western European allies have been loud and angry, and the prestige of every major leader in the western alliance is now publicly impaled on this escalating dispute. Diplomats on both sides of the Atlantic are driven to find an elusive compromise, save an extraordinary amount of face and avoid shattering the western alliance.

George P. Shultz, designated successor to Alexander M. Haig Jr., inherits a central American role in grappling with what a senior State Department official described last week as a "monstrous problem."

Inside the Reagan administration, there is reported agreement at the Cabinet level to seek urgent "damage limitation" talks with Western European policy makers while attempts to resolve the dispute are under way.

Reagan left himself in a vulnerable position Wednesday by claiming that his administration "largely eliminated" the "disarray with our European allies" inherited from the Carter administration. The next day British Prime Minister Margaret Thatcher, Reagan's strongest supporter in the western alliance, confounded his claim by telling the House of Commons that "it is wrong" for "one very powerful nation" to try to prevent the fulfillment of "existing contracts" in the pipeline uproar. The Europeans are now mounting legal challenges against the American sanctions, and the arguments already extend beyond equipment for the Siberian pipeline to disputes over new American duties on European steel exports to the United States, to tax subsidies for foreign subsidiaries of American corporations and to other trade-related controversies.

"It may be premature and exaggerated to talk in terms of a 'trade war,' " British Minister for Trade Peter Reese said in New York last week before pressing the British case with officials in Washington, "yet the danger of a series of measures and countermeasures--tit for tat--must be obvious to all."

Talk of a trade war among the western allies is a reversal of the original argument over East-West trade sanctions. West German Chancellor Helmut Schmidt and French President Francois Mitterrand warned last month that they will not join in a "trade war" with the Soviet Union, which they said would lead back to the era of Cold War. The United States denied that was its intention.

Nevertheless, the allied clash is more about ideology and East-West strategy than about pure economics, and the debate runs through the Reagan administration as much as it does through the western alliance.

A major American casualty in the dispute was departing Secretary of State Haig, who led and lost the argument inside the administration over imposing the pipeline sanctions.

Haig argued that Western Europe would reject overt economic pressure on the Soviet Union, opening a breach in allied ranks that could undercut all U.S. strategy for putting pressure on the Soviet Union to curb its buildup of military power.

His successor, Shultz, is noted for skill as a mediator, from his experience as secretary of labor, director of the Office of Management and Budget and ultimately secretary of the treasury in the Nixon administration. It will be several weeks, however, before Shultz goes through confirmation hearings, takes office and can begin to apply his negotiating talents to the trade dispute.

In the meantime, some Reagan administration officials are drawing encouragement from the European News Analysis News Analysis Economic Community's agreement last week to raise interest rates on financing western exports to the Soviet Union. The Reagan administration campaigned earlier for a curb on government-subsidized trade credits for the Soviet Union. That misled Western Europeans into believing that the United States, in turn, would drop its demands for sanctions on equipment for the 3,700-mile pipeline between the Soviet Union and Western Europe.

Some U.S. officials privately talk of ending the dispute by trading the pipeline sanctions for further restrictions on Soviet credits. Others, determined to exert maximum pressure on the Soviet Union, adamantly reject such a bargain.

The hard-liners insist that the only route open for lifting the pipeline sanctions is the easing of Soviet-supported repression in Poland. That was President Reagan's declared reason on June 18 for imposing the broadened ban on American-built or American-licensed pipeline equipment produced abroad.

Many administration officials concede privately, however, that the Polish situation was not the overriding reason for the pipeline sanctions. Rather, they say, it was the president's determination to inflict economic penalties on the Soviet Union. At his news conference Wednesday night, Reagan gave both rationales for his decision.

He said one purpose was to tighten the trade embargo on the Soviet Union until there is relief for its support of "the oppression that is going on of the people of Poland by their military government." The "second thing," he said, is that the Soviet Union "is very hard-pressed financially and economically today."

Its "massive military buildup," he said, will benefit from "$10-$12 billion a year in hard cash payments" which the Kremlin would obtain from natural gas delivered to Western Europe in the pipeline.

The Western European governments contend that the pipeline will be built in any case, and reject the argument that trade can be used as an effective weapon to constrain Soviet expenditures on its vital security interests. Many U.S. experts agree, but American officials are deeply divided on this issue. As Haig's resignation and the president's decision clearly illustrated, the predominant weight inside the Reagan administration is on the pro-sanctions side.

White House national security adviser William P. Clark has stated that it is administration policy to "force our principal adversary, the Soviet Union, to bear the brunt of its economic shortcomings." Defense Secretary Caspar W. Weinberger, Haig's principal antagonist on this subject and many others, was the first to enunciate that policy in the earliest days of the administration.

The argument is bound to recur in any effort to reach a compromise on this issue with Western European governments. Furthermore, any improvement in the situation in Poland could provide a rationale for a change in the American position. There is therefore high interest in Washington and throughout the Atlantic alliance in the attitude and compromise skills that Shultz will bring into the bruising debate.

There are several parallels in the Haig and Shultz positions. According to Haig's associates, his dominant reason for reproaching the administration in his resignation for a lack of "consistency, clarity and steadiness of purpose," was his complaint over shifting U.S. positions on sanctions against the Soviet Union.

Shultz has been committed for years to the need for consistency in American policy abroad. In his last major public address, in London in October, 1981, Shultz stressed the need for "giving confidence to ourselves and our partners in the predictability of our behavior and the consistency of our purpose."

Shultz is also on record as a long-standing opponent of the use of trade as a political weapon. In the early 1970s, as an advocate of detente in the Nixon administration and a strong supporter of Henry A. Kissinger's strategy, Shultz opposed the use of trade sanctions to exert pressure on the Soviet Union to relax its emigration restrictions.

As a champion of broader American-Soviet trade, Shultz twice went to Moscow as treasury secretary in attempts to resolve the emigration dispute with Soviet leader Leonid Brezhnev. Those efforts were blocked by congressional insistence on imposing a trade-emigration link in the 1974 U.S. Trade Act.

Kissinger, however, has since concluded that the use of trade to influence Soviet global behavior was doomed to fail. Shultz may have reached the same conclusion. Shultz presumably accepts the policy of the administration he is entering, an administration that scorns what it calls the "illusion of detente" and openly invokes trade as leverage against the Soviet Union. His specific views on trade sanctions as a weapon, however, are unknown, even among many former associates.

As secretary of state, Shultz will be in an entirely different relationship with Haig's chief opponent in the administration, Weinberger. Weinberger was a subordinate of Shultz in government during the Nixon years, and later served under Shultz in the powerful Bechtel engineering and construction conglomerate, which Shultz headed as president until last month.

When Shultz directed OMB, Weinberger was his deputy and then his successor after Shultz went to the treasury. Shultz left the government in 1974 to join the Bechtel Group, where he became president in 1975; Weinberger joined Bechtel that year as a vice president.

An intriguing question inside the bureaucracy, therefore, is whether the Shultz-Weinberger friendship will transcend the normal State-Defense competition on many issues of government policy, including the present deep disagreement between the two departments on trade sanctions aimed at the Soviet Union.