Soviet leader Yuri V. Andropov's proposal for limiting medium-range nuclear weapons in Europe is the first public act of a deadly dance to be played out over the next year by the United States, its western European allies, and the Soviet Union.

For those anxious to reduce the danger of nuclear war and yet preserve a balance of power, it is a dance that is likely to be increasingly difficult to understand because the moves are technically complicated and official statements on both sides may be deceptive.

Yesterday, for example, U.S. arms control chief Eugene V. Rostow revealed that U.S. and Soviet negotiators had informally discussed last year a compromise reduction of European-based nuclear missiles that apparently differed from the official public position of the Reagan administration.

The United States and the Soviet Union are conducting two negotiations at the same time in Geneva. One, known as the START talks, is aimed at reducing the intercontinental-range ballistic missiles and bombers that both sides have based by the thousands in their homelands.

The other, known as the INF talks, deals with intermediate-range nuclear missiles based on both sides in Europe.

Although the connection is not yet being made publicly, some officials have said that other highly charged issues -- including whether the Bulgarians and ultimately the Soviets had any role in or knowledge of the attempted assassination of Pope John Paul II last year -- may spill over into calculations of western public opinion about Soviet intentions.

The main emphasis of Andropov's speech and the focus of the negotiations dance at the moment is the question of missiles based in Europe, where public opinion is a key objective of both U.S. and Soviet negotiating moves.

The Soviets already have 333 new SS20 missiles, each carrying three warheads, targeted on Europe and Asia, plus 300 older missiles. The NATO alliance plans to deploy, beginning a year from now, 572 new American-built Pershing II and cruise missiles to offset the Soviet threat unless agreement on mutual reductions can be achieved at Geneva.

At the INF talks, President Reagan has made a "zero-option" proposal for the Soviets to scrap all 600 or so of their intermediate-range missiles and the United States to cancel deployment of its new Pershing II and cruise missiles. The Soviets rejected it.

In his speech Tuesday, Andropov proposed reducing the number of SS20s in the European portion of the Soviet Union to 162 missiles -- the same number that Britain and France now have in their independent nuclear forces -- in return for U.S. cancellation of its new missile deployment.

The Soviets argue that their nuclear missiles targeted on western Europe already are balanced by the British and French forces plus American bombers based in western Europe that could attack the Soviet Union with nuclear weapons.

But the United States, France, Britain and the NATO alliance reject this. They argue that there are no equivalent U.S. missiles to match the SS20s and that the Soviets have more bomb-carrying planes than the NATO allies in Europe.

The British and French also consider their nuclear weapons, nearly all of which are based on submarines, to be sovereign, outside NATO control and independent deterrents to Soviet attack on their own countries.

Andropov did not say whether he would destroy the SS20s he proposes to withdraw--a key point because they are mobile and could be moved back to their original positions.

Even if moved east of the Ural mountains, the SS20s, with their range of 3,000 miles, could reach much of western Europe. Nor did Andropov indicate whether he would remove any of the 100 SS20s based in the Asian portions of the Soviet Union. The State Department, therefore, rejected Andropov's speech within hours as a public relations ploy meant only to influence western opinion.

There are some in government, however, who say they believe that, while the Andropov proposal is not acceptable in its present form, it does reflect a willingness by the new Soviet leader to negotiate seriously.

In their view, by raising publicly the prospect of reducing Soviet missiles, Andropov may have been indicating that a compromise can be negotiated. This is a minority view, but it reflects differences within the Reagan administration over what to do next in Geneva.

Nevertheless, even those holding this more hopeful view said they felt it was necessary for the State Department to immediately reject the Andropov proposal because any impression that it was under consideration could create demands in Europe for delaying deployment of the new U.S. missiles.

The widespread view within the administration and NATO is that the Soviets will not offer any major compromise until they are convinced the West will otherwise go ahead with the deployment on schedule.

It also is widely assumed that the Soviets are determined to prevent the Pershing II, in particular, from being deployed because it could hit Soviet targets from West Germany with virtually no warning time.

Ideally, according to alliance analysts, the Soviets want to prevent deployment of any new U.S. missiles to poke a hole in the psychological umbrella of nuclear protection that the United States has provided western Europe for 30 years, thus weakening alliance ties.

But some U.S. officials say they believe it may be possible to work out a compromise at Geneva that would leave the two sides with an equal number of new missiles but at much lower levels than now planned.

Others, who are most worried about European public opinion, say they believe the United States should try to get the largest reduction possible by Moscow in return for complete cancellation of NATO deployment of the new U.S. missiles.

If such a cut were big enough, they point out, it could reduce Soviet missile warheads to about the same number as there were on the older Soviet missiles before the SS20 buildup began.

While the zero option remains the official and predominant Reagan administration position, officials said these alternative views are being voiced within the administration and no decision has been made on whether the American position will be changed before negotiators return to Geneva for resumption of the INF talks in late January and and the START talks in early February.

Yesterday, Rostow referred to Andropov's speech as "profoundly disappointing" because, in his view, it rehashed old Soviet ideas already answered by Washington and showed Moscow to be interested only in supremacy in Europe.

Rostow revealed, however, that what he called "a generally promising compromise initiative developed during last summer . . . an initiative the United States was willing to explore . . . was turned down flatly by the Soviet Union in September."

Although Rostow, in a telephone interview, would provide no details, his remarks, according to other officials, confirm that the United States has held secret and informal discussions with the Soviets on a compromise between the zero option and the Soviet solutions. Officials said privately that this involved equal but lower levels of missiles.

Word of these secret discussions first leaked out in Paris in October and has since been referred to in numerous accounts of the negotiations, but had not been confirmed officially before now. The White House reportedly rebuked the U.S. negotiators for engaging in these discussions.

Andropov, in his speech, also publicly confirmed that Moscow was willing to negotiate a 25 percent reduction of its intercontinental missiles and bombers in the START negotiations.

Georgi Korniyenko, a deputy foreign minister, had confirmed the same proposal last month in an interview with Newsweek magazine. What Andropov did not say is that Moscow also has tied this to cancellation of deployment of the Pershing II and cruise missiles in Europe.

This Soviet proposal is not new and is far less than the much deeper reduction proposed by Reagan. Nevertheless, Reagan has called the Soviet START position "a serious one" and said "there's no question we are heading in the right direction."

What the president apparently means is that the Soviets appear to have accepted the idea of significant reductions in nuclear arsenals, although the White House wants them to be much deeper.

The difference between START and INF, however, is that on the strategic issue, both countries already have huge arsenals aimed at each other from across the oceans and poles and thus, while an agreement is sought, the two sides are working from a rough balance.

In the European-based missile talks, the Soviets have a clearly larger force in place that NATO wants to negotiate away or match. The problem is that matching Moscow raises the prospect for some Europeans that their homelands might become the nuclear battlefield in any future war, which makes the issues emotional and immediate.