The successful outcome of U.S.-Soviet talks in Geneva last week was the product of a dramatically different Reagan administration policy-making process and a Soviet decision in advance to return to full-scale arms negotiations, according to members of the team that accompanied Secretary of State George P. Shultz.

What President Reagan called "American diplomacy at its best" seemed to break the deadlock at home and abroad that frustrated U.S. arms diplomacy from 1981 to 1984. Officials cautioned, however, that a successful "first step" in Geneva does not mean that the upcoming nuclear arms and space negotiations will produce early or easy agreements.

"We realized that the differences were papered over rather than really bridged and that there is a long road ahead," said a U.S. participant who declined to be named. "But talking with the Soviets is better than not talking . . . . We accomplished what we set out to do."

Interviews late last week with Geneva participants from the State and Defense departments, National Security Council and Arms Control and Disarmament Agency disclosed a general optimism about curbing the unbridled interagency infighting that marked arms policy during most of Reagan's first term.

The same officials were notably less optimistic, however, that the accommodating posture of Soviet Foreign Minister Andrei A. Gromyko at Geneva will set a pattern for the three-part substantive negotiations -- expected to begin in March -- on strategic arms, medium-range arms and "preventing an arms race in space."

On the domestic front, three factors in policy-making were cited:

* Reagan's "much more involved" role in the 4 1/2 weeks between his first National Security Council meeting Nov. 30 in preparation for Geneva and the signing of an unusually detailed and lengthy National Security Decision Memorandum on New Year's Day setting forth the negotiating positions.

Some officials insist that Reagan in his first four years was not as remote as depicted by such incidents as his admitted failure to understand that the Soviets rely primarily on land-based missiles, and Secretary of Defense Caspar W. Weinberger's use of a cartoon to sell Reagan on a basing mode for the MX missile.

All of those interviewed made the point, however, that Reagan was engaged this time to a much greater degree.

The president participated personally in five meetings in the White House Situation Room with his top Cabinet and arms advisers in the 4 1/2 weeks, according to officials. These intimate meetings were attended only by eight top officials plus four or five senior aides, making possible a greater degree of frankness and flexibility than earlier Cabinet Room sessions with large numbers of White House and departmental officials lining the walls and taking notes.

"At one point we finished a 17-page paper at 10 p.m. on a Saturday night, and by Monday morning Reagan had read it and made a large number of marginal notations," said a participant.

Reagan read "hundreds of pages" of reports on various issues, said another official in a position to make a count. Still another official emphasized that this time "the mix of paper and discussion shifted in favor of discussion," especially where Reagan was concerned.

* The influence of Paul H. Nitze, as special arms adviser to Shultz, in narrowing the gap between contending positions.

Because of Nitze, "The advice being given to Shultz was much better than before and the difference between Shultz and others was much narrower," said an arms control agency official. A Defense official added, "Nitze did not always agree with and invariably carried to a deeper analytical level the standard State Department view."

Shultz is reported by participants to have backed away from previous State Department positions favoring a three-year moratorium on antisatellite testing and an early "SALT II 1/2" accord modestly reducing U.S. and Soviet strategic weapons levels.

While saying they cannot be sure what changed Shultz' mind, insiders attribute the shift to the influence of Nitze, who is believed to be skeptical of the early State positions.

* Shultz' ability as a consensus builder and delegation leader, aided by national security affairs adviser Robert C. McFarlane.

McFarlane presided over twice-weekly meetings of second-level officials who often used to battle among themselves. This time, the knowledge that their bosses were compromising in higher-level meetings is believed to have muted disputes in this group.

Shultz' decision in early December to invite this entire group to Geneva was opposed by some within the administration because of fears that it would lead to a fiasco of arguments and leaks. It didn't turn out that way, in part because nearly everyone felt he had a chance to participate fully.

Kenneth L. Adelman, director of the arms control agency, said, "Shultz as head of the delegation really brought out the best in each of us." Assistant Defense Secretary Richard Perle said "a cohesiveness, a corporate sense of being part of a whole" developed at Geneva, "due primarily to the way Shultz managed the situation."

Following Reagan's detailed decision memorandum of Jan. 1, Shultz convened a series of meetings late on Friday, Jan. 4, and at breakfast Jan. 5 for key administration decision-makers and members of the big Geneva team.

Shultz, McFarlane and their delegation took off Saturday night for Geneva in an Air Force plane, dubbed by a skeptical reporter "the ship of feuds."

En route, officials from each of the sometimes-warring agencies took turns studying what they called "the book," the proposed "talking points" for Shultz' initial presentation to Gromyko.

Shultz solicited the comments of each member in a Sunday afternoon meeting in "the bubble," a secure room at the U.S. Mission in Geneva. He met again with members of his backup team Monday morning just before going to the Soviet Mission.

Joining Shultz in the sessions with Gromyko was a small group: McFarlane, Nitze, U.S. ambassador to Moscow Arthur Hartman, National Security Council aide Jack Matlock as note-taker, and an interpreter.

After each session, McFarlane briefed the larger group in detail, Hartman prepared a one-page memorandum for Reagan and Matlock worked on a memorandum of the conversation. Then the large group would meet Shultz and the other negotiators to discuss tactics for the next session.

During the crucial final bargaining session with Gromyko at the U.S. Mission Tuesday afternoon and evening, Shultz and McFarlane kept the large group informed of developments on almost a play-by-play basis.

When all members of the U.S. team left the main negotiating room to permit separate U.S. and Soviet caucuses, Shultz jokingly told Gromyko that, because the Soviet minister was taking over a U.S. Embassy room, "we'll have to call it 'the Gromyko room.' "

At Shultz' instructions, a State Department secretary posted a prominent sign, "Andrei A. Gromyko Room," to the amusement of the veteran Soviet minister.

The other basic element behind last week's outcome was the Soviet willingness to compromise and to abandon most of Moscow's earlier conditions for resuming arms-control negotiations.

Unlike some previous meetings, there was no advance agreement through diplomatic channels guaranteeing success. The only points established through contacts in the weeks before the talks, according to State Department sources, were that both sides favored eventual "radical reductions" in arms and that "an organic relationship" exists between offensive military power and potential defensive power such as Reagan's "Star Wars" plan to destroy incoming missiles.

The U.S. team anticipated that Gromyko would push for a moratorium, or temporary halt, in antisatellite and antimissile testing as the Soviets did last summer in proposing space weapons talks.

It had been decided in Washington not to agree to such moratoria at Geneva and not to suggest that the United States would accept them later. At the minimum, U.S. officials calculated, Shultz would have to explain Reagan's Sept. 24 U.N. General Assembly statement -- now out of line with U.S. thinking -- that the United States would consider "what measures of restraint such as testing moratoria both sides might take while negotiations proceed."

But Gromyko did not push the moratorium issue at Geneva and, according to several U.S. accounts, did not even bring it up.

U.S. negotiators anticipated a strong argument from Gromyko that testing and development of Star Wars components violates the 1972 Anti-Ballistic Missile (ABM) Treaty. In preparation for this argument, Shultz conferred at length with Thomas Graham Jr., general counsel of the Arms Control and Disarmament Agency. But in Geneva, U.S. sources said, Gromyko did not raise the issue.

Another set of time-consuming questions at the White House was what to reveal to Gromyko about potential U.S. positions if the strategic arms and medium-range arms negotiations were restarted. But in Geneva, according to U.S. sources, Gromyko did not ask.

The most contentious issues at Geneva revolved about the bargaining framework and objectives in the new category of space arms negotiations.

These issues mostly were "resolved" with vague language that leaves room for arguments in future negotiations.

Gromyko's willingness to be accommodating in the face of relatively unyielding U.S. positions has left the impression with several officials -- and reinforced an earlier belief on the part of others -- that the Kremlin decided well in advance of Geneva to return to arms negotiations with the United States.

Despite initial positive signs, the Geneva outcome was plunged into doubt during the final Tuesday afternoon session when Gromyko suddenly proposed to announce that the two sides would have to meet again, perhaps in Moscow, to resolve their differences.

The U.S. team was sharply divided about whether this was a negotiating ploy or a serious move by Gromyko to depart without positive results. After a strong argument by Shultz to the Soviet minister, it turned out to be the former. Gromyko backed down.