When President Reagan overpowered congressional Democrats last year and forced them to shelve most of their arms-control proposals on the eve of the U.S.-Soviet summit in Reykjavik, he got some friendly but blunt advice from Sen. Sam Nunn (D-Ga.).

Pay some heed to the Democrats' proposals or "the water will be over the top deck and it will be every rat for himself," warned Nunn, who had supported the administration in fending off legislative constraints on such issues as compliance with the unratified 1979 Strategic Arms Limitation Treaty (SALT II).

The advice went unheeded this year as the administration, despite waning influence on Capitol Hill in the wake of the Iran-contra scandal and the Democratic takeover of the Senate, took one action after another that defied the views of a clear majority of both houses on key arms issues.

It broke through SALT II weapons limits, threw a new roadblock in the way of ratification of two nuclear-explosion limitation treaties and advocated a broad interpretation of the 1972 Antiballistic Missile (ABM) Treaty to permit expanded testing of the Strategic Defense Initiative (SDI) -- enraging Democrats on all counts.

Brushing aside past differences, Democrats pulled together and challenged the administration with new-found zeal, refusing to blink at charges that they were playing into the Kremlin's hand and threatening even to hold funding for SDI as hostage for a compromise.

Nunn's warning had become prophecy. The deck was awash; even the conservative Nunn, now chairman of the Senate Armed Services Committee, had gone overboard.

As a result, the White House and Congress were locked in another arms-control stalemate, seemingly more intractible than the one that preceded it, as yet another summit approached. But this time, as the administration was putting finishing touches on plans for Reagan's meeting here this week with Soviet leader Mikhail Gorbachev, it was the White House's turn to back off from a confrontation.

In private meetings over several weeks in October and early November, then-national security adviser Frank C. Carlucci, sometimes accompanied by White House chief of staff Howard H. Baker Jr., met with senior members of the House and Senate armed services committees to work out a deal along lines that the administration refused even to consider the year before.

In public, Nunn and other Democratic senators had already sketched areas of possible compromise during the Senate's long, stormy and filibuster-plagued debate on the fiscal 1988 defense authorization bill, to which the arms-controls provisions were being attached. In private, Nunn and Carlucci came to an understanding that they would talk -- no guarantees, just talk -- after the Senate completed its work.

When he figured the time was right, Nunn got Sen. John W. Warner (Va.), ranking Republican on Armed Services, to arrange a meeting with Carlucci, which led to another meeting that finally brought House, Senate and White House negotiators into the same room.

"I was amazed," said House Armed Services Committee Chairman Les Aspin (D-Wis.). "I thought they'd use it {the approach of the Reagan-Gorbachev summit} to put us up against the wall again rather than the other way around, rather than using the impending summit to cut a deal."

But, from the first three-way meeting, Aspin said, it was clear that the administration wanted -- and would get -- an accommodation. "No sooner than we got in the room we were talking serious discussion about how to resolve the whole damn thing," he added.

"We weren't getting anywhere and neither were they," said a senior administration official. "It was clear the confrontational approach wasn't working. We could have vetoed the bill, but it would have been right back before us in some form. We didn't want restrictions on everything {meaning ABM and SALT II} going into the negotiations and summit with the Soviets. That was probably the most important thing. We also didn't want cuts in SDI."

As the compromise turned out, the administration would be forced to dismantle a nuclear-armed Poseidon submarine to stay within existing strategic weapons levels, slightly above those allowed by SALT II, and stick with an earlier administration testing schedule for SDI that is in line with a restrictive reading of the ABM pact. The administration was particularly reluctant about the Poseidon/SALT II deal, never specifically bargaining on that point, although it had to accept the dismantling provision as the price for an ABM/SDI deal and a defense bill for the year.

In exchange, Congress agreed to drop its earlier demand for language in the defense bill that had the effect of imposing the treaty constraints by statute, which the administration regarded as particularly offensive.

In addition, the House would drop its insistence on the nuclear test ban in exchange for continuation of the current moratorium on tests of antisatellite weapons against objects in space.

In one sense, it was little more than a cease-fire, with face-saving for both sides. The administration could accurately claim that Congress backed away from legislating treaty constraints, while Congress could just as correctly say it had prevented the administration from new treaty departures by denying funds for weapons or testing programs that exceeded those same constraints.

It did not resolve the underlying disputes involving SALT II compliance and ABM interpretation; it ducked the even more fundamental issue of the balance of power between Congress and the White House on war-and-peace issues.

But the fact that agreement was reached indicates the degree to which the politics of arms control in Washington has been reshaped over the past year.

Whipsawed by events ranging from senatorial elections to the Iran-contra scandal, the arms-control debate has become more polarized, more partisan. But it has also become more evenly balanced, posing a risk of stalemate that turned out to be scary enough to force compromise rather than continued confrontation, according to players on both sides.

The Democrats' recapture of the Senate in last year's elections was the most obvious factor in the shift. Freshman Democrats made the difference on the key issues, reversing the Senate's position on SALT II and providing a bigger-than-expected margin on the ABM vote.

Moreover, Nunn's elevation to the Armed Services chairmanship, coinciding with his move toward the mainstream Democratic position on arms control, gave the Democrats a powerful new voice -- the voice of a hawk -- on the ABM issue in particular.

It also provided a bridge over a major divide in the party as Democratic liberals, often frustrated in the past by conservative defections, worked with Nunn -- or at least without his opposition -- to push their causes. The ABM proposal was originally advanced by Sen. Carl Levin (D-Mich.), who worked hand-in-glove with Nunn to get it passed. As Sen. Dale Bumpers (D-Ark.) pushed the SALT II proposal, Nunn, who had previously opposed such moves, shifted into neutral and eventually voted for it. In a rare display of unity, the Democrats lost only one of their own (Ernest F. Hollings of South Carolina) on the ABM issue; they lost five on the final SALT II vote.

A different dynamic was at work among House Democrats. Aspin had almost lost his chairmanship earlier this year when liberals rebelled at some of his previous deal-cutting, and he was under heavy pressure to deliver for them this time.

Liberal arms-control advocates such as Rep. Edward J. Markey (D-Mass.) were appointed as special conferees on the arms issues and held out to the end for concessions on verification plans for a nuclear test-ban agreement. They lost, but in the process fended off similar concessions sought by the administration that could have made it easier to resume antisatellite testing.

The significance of the Iran-contra affair was not just a waning of administration credibility and influence. More importantly, many lawmakers said, in tapping Carlucci to straighten out national security operations, the president chose someone who understands Congress and can deal with it. Lawmakers who had given up trying to deal with Defense Secretary Caspar W. Weinberger, whom Carlucci succeeded last month, found Carlucci to be a tough but willing negotiator.

"His personality as an individual who was willing to listen to the other side began to emerge in contrast to Secretary Weinberger -- and I don't say this critically -- who didn't have in him the willingness to listen," said Warner, who had also pushed for compromise, sometimes at odds with hard-liners in his party. "Frank Carlucci," Warner added, "has the art of listening."

In the final analysis, however, it appeared from interviews with all principals that the administration lost its chance to force Congress into another arms-control retreat this year before the fight even started.

Democrats felt betrayed by the SALT II breakout and by other administration moves that they regarded as a flouting of the 1987 understandings. Nunn was especially angered because he had gone against fellow Democrats to back the president, only to be confronted with what he calls a "breach of faith" when the administration demanded Senate approval of verification rules for the two nuclear-explosion treaties as a vote on the treaties themselves.

This time the Democrats were prepared to play their own summit card. Nunn was talking about continuing SDI at current levels until early next year and tying future funding to acceptance by the administration of Congress' conditions. Aspin was toying with a trap-door scheme whereby $3.9 billion would be allocated for SDI research, a modest increase over current spending, with the proviso that only $1 billion could be spent unless a compromise defense bill was passed and signed.

Moreover, there would be no defense authorization for this year -- "no certainty in contracting, no nothing," throwing "everything into turmoil," Nunn said. No more warnings about water over the deck. This time, said Nunn, there would be "blood on the floor."