The recent Soviet offer to limit the number of its intermediate-range nuclear missiles in Europe had its roots in informal discussions last summer between American negotiator Paul H. Nitze and his Soviet counterpart, Yuli A. Kvitsinsky, at the Geneva arms control talks, according to informed sources.

After Moscow and Washington rejected a new negotiating framework put together by Nitze and Kvitsinsky to try to achieve a compromise limit on the number of Soviet and U.S. nuclear missiles based in Europe, Kvitsinsky in November informally discussed a new Soviet proposal since made public by Soviet leader Yuri V. Andropov.

But Nitze, who was sent back to Geneva in September with instructions sharply limiting his freedom to bargain informally with Kvitsinsky, had to use a visiting senator, Gary Hart (D-Colo.), to help explore some of the details of the new Soviet plan and was not given the authority to follow up.

Nitze has decided that he does not want to be placed in the same awkward position when the talks resume next week, the sources said, and intended to ask President Reagan at a meeting this week for greater flexibility in responding to the Soviets when he returns to Geneva. The White House said yesterday that Reagan may meet with Nitze later this week.

A State Department spokesman said yesterday that Nitze "has always been authorized to explore any flexibility in the Soviet position." But the spokesman, when questioned, refused to say whether Nitze could indicate any flexibility in the American position.

The first official confirmation of the informal talks between Nitze and Kvitsinsky last summer was published in The Washington Post on Dec. 23, after Andropov made public the new Soviet proposal.

Eugene V. Rostow, then the Reagan administration's arms control director, said in an interview that the Geneva negotiators had discussed "a generally promising compromise initiative developed during last summer . . . an initiative the United States was willing to explore . . . . It was turned down flatly by the Soviet Union in September."

Last weekend the story was revived by some officials who cited this incident as a cause for last week's firing of Rostow as director of the Arms Control and Disarmament Agency. Those officials said Rostow had been aware of Nitze's informal negotiations and initially defended him when White House national security affairs adviser William P. Clark wanted to discipline him.

However, most government officials, including Reagan, have maintained that Rostow's firing stemmed from personal and managerial differences rather than the Nitze incident.

According to informed sources, Nitze and Kvitsinsky, two experienced negotiators, reached an extraordinary agreement between themselves last July on a new "package" approach to the negotiations on limiting nuclear missiles in Europe, which appeared to them to have bogged down after two rounds of discussion.

The package, these sources said, included the following elements.

The Soviets would freeze the number of their intermediate-range SS20 nuclear missiles based in the Far East and reduce the number of SS20s in central and western Russia that are aimed at western European targets.

The United States would not deploy the Pershing II, the missile the Soviets most feared because it could hit Soviet territory within eight minutes after launch from planned bases in West Germany. The United States also would limit the number of ground-launched cruise missiles stationed in Europe, depending on the number of Soviet European-based SS20s.

No number was set, but the United States would be left with more warheads and fewer launchers because an SS20 carries three nuclear warheads while a cruise missile launcher will fire four separate nuclear missiles.

There would be limitations on the number of nuclear-capable aircraft in Europe with the goal of equalizing numbers for both the NATO and Warsaw Pact countries, and particularly American and Soviet planes.

This package strayed considerably from the initial negotiating positions put forward by the two countries.

The original Soviet position called for a reduction in NATO and Warsaw Pact nuclear systems, both aircraft and ground and sub-launched missiles, to 300 by 1990. This would prevent the United States from deploying either the Pershing II or cruise missiles and force it to reduce the number of its nuclear-capable aircraft.

The U.S. position was to negotiate only about missiles and to pursue a "zero option," under which the United States would not deploy any of the planned 108 Pershing or 464 cruise missiles if the Soviets agreed to destroy not only their roughly 300 older SS4 and SS5 missiles but also all of their 300 new, deployed SS20s.

The U.S. and Soviet negotiators, sources said, tried to narrow the wide gap between the two positions by establishing a logical framework for missiles and bombers without setting specific numbers.

For example, the Soviets wanted to include British and French nuclear missile systems within the American totals, while the United States wanted to eliminate all SS20s in the Soviet arsenal, even those stationed in the Far East and targeted on China.

The Nitze-Kvitsinsky approach, one source said, was to freeze the number of SS20s in the Far East and appear to equate them to the British and French systems. That left the opportunity to reach some parity between U.S. and Soviet missile warheads in the European area.

In July, after Nitze's initiative became known in Washington, some Reagan administration officials argued that the arms control expert had exceeded his authority and should be reprimanded. He was not. But when Nitze, 76, returned to Geneva in September for the third round of negotiations, his instructions sharply limited his authority to engage in such exploratory discussions.

He felt so tied by his instructions that he used Hart one day as an intermediary with Kvitsinsky to explore hints of a new Soviet negotiating position. As the three lunched together, the two negotiators posed questions and offered answers to each other through Hart, who later said he "served as a hollow log."

Although he cabled the informal new Soviet positions to Washington, Nitze was not given any authority to follow them up, according to sources.

Much of the Soviet proposal since made public by Andropov appears to have roots in the Nitze-Kvitsinsky discussions last summer, including separate limits for missiles and aircraft, rough parity on numbers of missile warheads rather than launchers, and the destruction of some SS20s.