The negotiations scheduled to begin in Geneva Nov. 30 between Washington and Moscow on limiting nuclear arms based in Europe are apt to be the most difficult and complicated in the roughly 20-year history of superpower efforts at controlling atomic weaponry.

This is so because the United States and the Soviet Union approach these talks with vastly different goals in mind and the weapons involved are smaller, more numerous, more mobile, harder to find and thus harder to count than those that have been the main subject of arms control talks in the past 10 years.

During the 1970s, Moscow and Washington engaged in strategic arms limitation talks (SALT) that dealt with limiting the big ocean-spanning, 6,000-mile-range land-based missiles, bombers and missile-firing submarines with which each could strike directly at the other's homeland.

But the forthcoming Geneva talks will deal with what are called theater nuclear forces or TNF, meaning hundreds or even thousands of medium- and intermediate-range missiles and aircraft that can shoot or fly from a few hundred to perhaps 2,500 miles on a battlefield that could cover all of Europe, including parts of the Soviet Union.

These talks grow out of a momentous and politically controversial decision by the North Atlantic Treaty Organization in December, 1979, to deploy, beginning late in 1983, 572 new U.S.-built Pershing II and cruise missiles in West Germany, Britain, Italy and possibly Belgium and the Netherlands.

These missiles were meant to offset a substantial arsenal of new Soviet SS20 2,500-mile-range missiles already in place and targeted on NATO bases. The intention was to plug a gap in the ability of NATO to deter a Soviet attack in the first place by making it clear to the Soviets that if they attacked Western Europe and not the United States, they would not escape retaliation on their own territory from West European bases with American-controlled weapons. Both the Pershing and cruise missiles have ranges in excess of 1,000 miles and could reach Soviet territory.

At the same time, many West European leaders, mindful of their political left wings and of irritating their big Russian neighbor too severely, insisted that the United States also negotiate with Moscow as soon as possible to try to limit such deployments on both sides to as few missiles as possible.

Thus, the United States goes into these talks convinced of the absolute need to modernize NATO's arsenal yet with a commitment to talk, while the Soviets go in with a top priority of stopping the U.S. program before the first new missile is installed.

So while the talks represent a hopeful sign to those advocating arms control and represent the first such contact with Moscow by the Reagan administration, there are huge obstacles in the way of agreement, and even very little agreement about what is to be discussed.

For example, Reagan administration officials say they want the initial negotiations limited to the SS20 and some 350 older SS4 and SS5 Soviet intermediate-range missiles and, on the American side, the Pershing II and cruise missiles. In other words, this means only land-based missiles.

The Soviets already have about 250 of the triple-warhead SS20 missiles deployed, but about 75 of these are said to be based in the eastern U.S.S.R. , presumably aimed at China rather than Europe. But because the SS20 is mobile and can be moved easily by truck, Washington wants the SS20 limitation to be "global," meaning all are counted in the overall balance of forces targeted on Europe.

Washington has not yet decided, officials say, how to deal in the talks with a variety of even newer Soviet SS21, SS22 and SS23 missiles which may soon be deployed with ranges from 65 to 1,000 miles. Nor has it decided whether it will propose that Moscow's 4,300-mile-plus-range Backfire bomber be included in the TNF talks or any resumption of SALT discussions.

Moscow, however, views the situation much differently. The Soviets say that the new NATO deployments are a fundamental effort by the West to tip the strategic balance because the Pershing and cruise can reach Soviet soil while the Soviet weapons in the TNF talks cannot reach the U.S. homeland.

And Moscow has already made clear that it doesn't want the talks limited to land-based missiles but rather wants all U.S. "forward based systems," meaning American fighters and bombers based in Europe, Britain and aboard aircraft carriers put on the bargaining table since some of them also can reach Soviet territory.

The United States, officials say, is still studying the question of if and when such planes would be included. But U.S. officials stress that in attempting to set equal, initial ceilings on European-based arms, they are talking about missiles and not aircraft. Furthermore, if the negotiations ever get around to aircraft, the United States is prepared to argue that all Soviet aircraft that can reach Western Europe, including thousands based in Poland, East Germany and Czechoslovakia, should also be put on the table.

The United States will also insist that whatever agreements are reached be verifiable. But officials acknowledge that, because these weapons are so small and numerous, this will be more demanding than in any past arms control agreements. So among other things there is no decision yet as to whether on-site inspection, in which observers actually go into both countries to verify compliance, will be demanded, something else that would require a major political breakthrough.

When NATO agreed to the arm-and-talk plan in 1979, it was stated that the talks go ahead within the framework of the SALT strategic arms process. But as of now, officials say it also is not clear what relation TNF will have to SALT.

On the one hand, the Reagan administration has made clear it does not like past SALT agreements and will not accept the SALT II pact signed by former president Carter and Soviet leader Leonid Brezhnev in 1979.

On the other hand, officials acknowledge that TNF limits are largely meaningless without limits on strategic weapons because strategic weapons, if there are enough of them, can be used to hit all the theater targets as well.

Similarly, the administration is anxious that the new NATO missiles be perceived in Western Europe as part of "a seamless web," as officials put it, of American protection that runs from conventional forces through theater nuclear forces and up to the big strategic forces. In other words, the United States does not want the TNF talks to heighten the very perception that Moscow is seeking to implant in Europe, namely that the United States is preparing to fight a war in Europe that won't encompass the American homeland.