In Geneva next week, the Reagan administration and the Soviet Union begin an extremely difficult search for a measure of stability in their relationship, some middle ground between the glow of detente in the 1970s and the bristling hostility of the early 1980s.

That goal is not listed on the schedule for Secretary of State George P. Shultz's Jan. 7-8 meeting with Soviet Foreign Minister Andrei A. Gromyko, but it is there between the lines. The negotiations they seek to launch in nuclear arms control are more complex and controversial than ever. U.S. and Soviet diplomats agree it will take unusual long-term resolve on both sides just to sustain the negotiating process in the constant friction of superpower rivalry.

Non-governmental American specialists also see a major obstacle to agreement in the administration's much-advertised internal split over Soviet strategy, a split that has not been resolved by the president.

Never before have the United States and the Soviet Union instructed their foreign ministers to produce an agenda encompassing "the whole range of questions concerning nuclear and outer-space arms." That sweeping objective was inspired not by optimism, however, but by failure; it is a catchall for past stalemates and new dilemmas.

The years 1981-84 were barren for nuclear arms control. Dual negotiations to limit intermediate-range and intercontinental nuclear weapons collapsed at the end of 1983 in Geneva. Nearly a year earlier, on March 23, 1983, President Reagan projected the space-based defense system dubbed "Star Wars," later named the Strategic Defense Initiative. By challenging the concept of nuclear deterrence, which is based primarily on offensive retaliatory weapons, Star Wars helped to throw the entire subject of nuclear arms control into widening arcs of controversy.

The blaze of world attention on the Shultz-Gromyko meeting bothers much more than it pleases diplomats, who see it as out of all proportion to anything next week's talks can achieve: at best, agreement on an agenda for subsequent arms negotiations. Stage-setting procedural issues, normally important only to specialists, suddenly have become front-page news.

Shultz was dismayed to hear in mid-December that U.S. television networks planned to send star "anchor teams" to Geneva, automatically raising public expectations for spectacular results. The Reagan administration, which initially was eager to dramatize its desire for arms control, anxiously began to caution against inflated expectations, which could put pressure on the United States in the bargaining at Geneva.

It is a maxim in diplomacy that serious negotiations take place in secret, and the more serious, the more secretive; but East-West diplomacy often flouts the rule.

Both sides have jockeyed intensively for public advantage in advance of the Shultz-Gromyko talks. Soviet Politburo member Mikhail S. Gorbachev in Britain last month, openly soliciting allied opposition to Star Wars, illustrated the weight being given to public diplomatic maneuvering. So did the counterattack by the Reagan administration in the president's meetings with British Prime Minister Margaret Thatcher and other western leaders, justifying Reagan's plan in its current research stage.

The Soviets fear being leapfrogged by the sort of nuclear technology that could turn up in Star Wars research. This ensures a continuing drive by the Kremlin to exploit widespread Western European apprehension about any shift in nuclear strategy. To Europeans, that strategy, with all its contradictions, has brought 40 years free of nuclear or conventional war between the major powers. It is not entirely coincidental, therefore, that the U.S. and Soviet delegations in Geneva each will be equipped to draw on the knowledge and memory of a senior official who participated in shaping that 40-year history. An Old Hand on Each Side

On the Soviet side, of course, it is the durable Gromyko, now at the peak of his political influence at 75, as deputy prime minister and foreign minister serving a new Soviet leader, Konstantin U. Chernenko. On the American side it is Paul H. Nitze, who will be 78 on Jan. 16, a special adviser to Shultz, and the most experienced American still active in nuclear negotiations.

Both men can see "new" problems in nuclear arms control as variations of problems that appeared in some form during the last four nuclear decades, sometimes with the two superpowers in positions opposite to those they now hold. Gromyko, who became Soviet ambassador in Washington at age 34, has dealt with 14 American secretaries of state starting with Cordell Hull, while Nitze has served Democratic and Republican administrations intermittently since 1940 as a policy-maker or senior negotiator.

In 1950, as head of the State Department's policy planning staff, Nitze was chief designer of a then-secret directive known as NSC 68, still regarded as the basic document on U.S. strategy in the Cold War.

The directive was a blueprint for expanding U.S. military power and hardening the American policy for "containment" of the Soviet Union by "a policy of calculated and gradual coercion" after it broke the U.S. atomic monopoly in 1949. NSC 68 was written in alarmist rhetoric to jolt the federal bureaucracy. Reagan could readily have drawn on it for his "evil empire" descriptions of the Soviet Union: it depicted Americans in "their deepest peril," confronted by "a slave state," a "despotic oligarchy" reaching for "world domination."

Nitze long ceased using such rhetoric, while Gromyko, in standard Soviet style, employs equal or stronger language in his current official writing. A foreword by Gromyko to a 1983 book entitled "Modern Diplomacy of Capitalist Powers," for example, illustrates that U.S. and Soviet indictments of each other's operating methods are exactly the reverse, with such characterizations by Gromyko as: "Needless to say, deception, blackmail and dictation, which have become the stock-in-trade of bourgeois diplomacy, are inapplicable in the practices of socialist diplomacy for reasons of principle and morality."

In Soviet diplomacy as well as ideology, it is a fundamental contention that the United States moderated its policies toward the Soviet Union only because it was "compelled" to do so by the growth of checkmating Soviet power, especially strategic nuclear might. The Soviet Union therefore attributes the nuclear "parity" or "equality" with the United States that it achieved at the end of the 1960s, and confirmed in the U.S.-Soviet detente accords in the early 1970s, solely to Soviet strength, which induced "greater realism" in Washington's policy.

Geneva has special significance in the two nation's conflicting perceptions of their nuclear-age history. Thirty years ago, the city was the site of the first postwar East-West summit meeting, the four-power conference of 1955.

At that point, two years before the Soviet Union's Sputnik breakthrough in space and missile technology, the United States had overwhelming nuclear superiority over the Soviet Union, which Secretary of State John Foster Dulles traded upon for his "brinkmanship" brand of diplomacy. Overruled by President Dwight D. Eisenhower on U.S. participation in the conference, Dulles set out, alternatively, to deny what the State Department listed as the prime Soviet goal in Geneva: to seek "moral and social equality" with the United States. 'An Austere Countenance'

Documents made public in recent years show that the State Department cautioned the president to avoid "social meetings" where he could be photographed with Soviet officials such as Nikita S. Khrushchev, who emerged at Geneva as the dominant Soviet leader succeeding Joseph Stalin. If that proved impossible, Eisenhower was advised to display "an austere countenance on occasions where photographing together is inevitable."

Dulles was even more troubled, "terribly worried," he confided to a colleague, that the British or French delegations at Geneva might "fall for some Soviet trick," or "accept some near disastrous compromise." But what "most worried" Dulles, he said, was "some slip of the president's," for Eisenhower, "so inclined to be humanly generous, to accept a superficial tactical smile as evidence of inner warmth, . . . might in a personal moment with the Russians accept a promise or a proposition at face value and upset the apple cart."

None of those calamities came to pass. Except for Gromyko, then a deputy foreign minister, and his boss, Stalinist-era Foreign Minister Vyacheslav M. Molotov, the Soviet officials were awkward novices on the world stage, with larger concerns and inhibitions of their own.

"We returned to Moscow from Geneva knowing that we hadn't achieved any concrete results" on nuclear arms control or any other issue, Khrushchev wrote in his memoirs. "But we were encouraged, realizing now that our enemies probably feared us as much as we feared them . . . . The Geneva meeting was an important breakthrough for us on the diplomatic front. We had established ourselves as able to hold our own in the international arena."

Eisenhower's great disappointment at Geneva was Soviet rejection of the "Open Skies" plan he unveiled, proposing that both superpowers open their territory to aerial inspection and exchange blueprints of military establishments. But Eisenhower was almost alone in his surprise, for the plan was drafted by specialists with the general expectation that it would be scorned; the Kremlin under czarist or communist rule always has placed the highest priority on Russian secrecy. The Soviet rebuff of 1955 became part of the internal U.S. rationale in 1956 to "open up" the Soviet Union unilaterally with American U2 "spy planes."

One Soviet history notes: "U.S. leaders sometimes had to give up their Cold War dogmas and agree to talks with Soviet leaders: at Geneva in 1955, at Camp David in 1959, in Vienna in 1961, and at Glassboro in 1967," but "these talks had a short-lived effect . . . because the American side was not yet prepared to accept in good faith the principle of peaceful coexistence as the basis for relations with the Soviet Union."

It was at Glassboro, N.J., that President Lyndon B. Johnson and Defense Secretary Robert S. McNamara tried in vain to convince Soviet Prime Minister Alexei N. Kosygin that the Soviet Union should abandon a nuclear strategy based on defense -- the early Soviet parallel for Reagan's Star Wars.

The arguments being used today on both sides of the current debate about defensive versus offensive nuclear deterrence, and the risk of an endless arms race, repeat almost verbatim the debates of the 1960s, except that they are in reverse and now extend to projections of exotic space-based defenses.

Not until Congress narrowly voted for an anti-ballistic missile system did the Soviet Union, which already had missile defenses around Moscow, shift course. In a 180-degree turnabout, the Kremlin made an anti-ballistic missile pact its priority, out of concern that it would be overrun in the technological race with the United States -- a fear of Soviet strategists then as now.

Negotiations between 1969 and 1972, known as the strategic arms limitation talks, or SALT I, with Henry A. Kissinger the Nixon administration's chief strategist, produced two fundamental accords: the Treaty on Limitation of Anti-Ballistic Missile Systems, and the companion agreement on Limitation of Strategic Offensive Arms.

For Soviet leader Leonid I. Brezhnev, however, the most glittering agreement signed with President Richard M. Nixon in May 1972 was a document that meant little to Americans: "The Basic Principles of Relations Between the U.S.S.R. and the U.S.A."

For Moscow, it represented not only acceptance of the Soviet concept of "peaceful coexistence" but also its long-sought "equal security" and "equality" in all fields. In fact, the document reflected a double misrepresentation: Soviet "peaceful coexistence" excluded any limitation on support of ideological warfare or "wars of national liberation," and the U.S. pledge of "equality" did not concede to the Soviet Union equal preeminence in world affairs, or an equal voice in every dispute.

The Nixon-Kissinger strategy, in any event, intended to rely primarily on a combination of "rewards and punishments" to induce Soviet restraint in its international conduct, or to penalize the lack of it. American expectations of Soviet behavior were so inflated by the Nixon administration, however, and so unfulfilled, that in the turmoil of the Watergate scandal and the Vietnam war, as Kissinger bitterly protested, U.S. strategy was denied both carrots and sticks. Crumbling of Detente

American-Soviet detente policy, launched in 1972, crumbled in stages between 1974 and 1979. The Soviet invasion of Afghanistan in late 1979 was the final blow, carrying down with it the last hope of the Carter administration for ratifying the intended successor to limitations on offensive weapons in the 1972 accords, known as SALT II, a product of seven years of effort by three administrations.

Even before the shock over Afghanistan, a decade of accumulated grievances about Soviet nuclear and global policy engulfed SALT II in Senate hearings, and one of the principal attackers was Nitze, who had helped to negotiate SALT I. A broadside of charges that the Soviet Union had violated its pledges, had achieved nuclear superiority, and had thrown the United States on the defensive globally, carried into the 1980 presidential campaign to help elect Ronald Reagan.

The Reagan administration, on taking office, returned to the 1950s NSC 68 prescription for building military strength as a prelude to any bargaining with the Soviet Union. In the administration's negotiations so far, Nitze, by contrast, has been in the unusual role of moderate, especially after the White House and the Kremlin in 1982 balked at his "walk in the woods" effort with a Soviet counterpart to break the deadlock over limiting European-based missiles.

The opportunity now recurs for another attempt to crack the nuclear impasse between the Reagan administration and the Kremlin. Just over a year ago that appeared improbable to many in Moscow, where the Soviet leadership was telling its citizens that were saying the two superpowers appeared to be on a collision course.

"Comrades," cautioned Grigory V. Romanov, the No. 3 man in the Soviet Politburo, on Nov. 5, 1983, "the international situation at present is white hot, thoroughly white hot."

That somewhat overstated tension has lessened, but it can never disappear while the United States and the Soviet Union remain nuclear rivals. Five years ago, former British ambassador to Washington Peter Jay wrote bluntly about the disappointment in store for those who envision an end to American-Soviet nuclear rivalry:

"The United States and the U.S.S.R. are doomed to watch one another like hawks, to negotiate constantly by day for strategic parity and to plot ceaselessly by night for strategic advantage. Since neither can or will feel fully confident unless its parity is more equal than the other side's parity, dynamic instability is inherent in the very static stability they seek."