We ought to have learned by now to distrust the signs of sudden improvement in relations between Washington and Moscow of the kind that we are now seeing on all sides, but hope springs eternal. When President Carter says, as he did in his speech to the United Nations yesterday, that the United States and the Soviet Union are within sight" of a new arms-control agreement, then it is obvious that a breakthrough of the kind that eluded Henry Kissinger for two years is in the making, if it has not been made already. How was it done?

All we can see on the surface so far is a change of atmosphere, but there is every reason to believe that substantive progress has been made in the private talks held recently between Soviet and U.S. officials. The ice did not really begin to move until Andrei Gromyko, the Soviet foreign minister, used his visit to the United Nations last month to convince the U.S. administration that Moscow was now willing to accept the basic negotiating context proposed by Carter, which it had rejected so vehemently earlier this year.

What Carter wanted to talk about was not just the limitation of strategic arms, but also "deep cuts" in the arsenals maintained by both sides. What Moscow wanted first was acceptance by Carter of the limits that had been agreed upon with Kissinger and President Ford in Vladivostok in 1974. The Soviet reaction to the "deep cuts" proposed by Secretary of State Cyrus Vance during his visit to Moscow in March was savage - as savage as the U.S. proposals were in Moscow's view.

Pravda denounced them as a demand for "unilateral disarmament" by the Soviet Union. It saw then as a "crude violation" of the Vladivostok agreement, and said they had been dictated to the Carter administration by the military-industrial complex. For a time it seemed that we were back on the brink of the cold war.

Carter's challenge to Moscow on human rights made the Kremlin suspect that the political survival of the Soviet regime, as well as its military strength, was being put in question by the new administration. In the West, many observers and even some friendly allies saw Carter's approach as a major "miscalculation" by a new, arrogant administration that had not yet learned the realities of international power, and they urged it to pull back before it was too late. But Carter, while moderating his language on human rights, was slow to make any concessions of substance.

Instead, he and his top aides sought to explain, in private meetings with journalists, that they wre following a deliberate but unprovocative strategy. After Vietnam and Watergate, the United States was perceived as being at a low ebb by its own people as well as by the Kremlin. The new administration was taking a firm line to restore the spirits of the American people, and also to discourage the Kremlin from trying to cash in on America's supposed weakness, as it had done in Angola.

It might take time for Soviet perceptions to change, and there were some risks in making the United States appear aggressive, but Carter was determined to persist. He wanted to give the Soviet Union the time it need to accommodate itself to the new situation, and he saw the Kremlin as going through a painful readjustment - but this was inevitable, because it had seen America in retreat, and had been caught unawares by its new self-assertiveness. Carter fully expected the difficulties in relations to continue for some months, but in the end, he insisted, Soviet perceptions would change and Moscow would return to a more cooperative attitude.

It looks a sif this is what has now happened, in large part, for the reasons given originally by Carter, as a result of the strategy worked out by his foreign policy adviser, Zbigniew Brzezinski. But it has not all been on way. The Soviet Union has made some concessions in SALT, but the United States has matched them, as Carter himself has said. The White House moved closer to the Kremlin on the Middle East, and this must have helped those Soviet leaders who wanted to resume active negotiations with the United States. They could now seek to persuade their more recalcitrant colleagues to move more quickly.

Nor is the strategic-arms agreement in the bag yet, and both sides have been careful to emphasize that some important issues still remain to be resolved. But while neither side would say for publication what has been agreed upon and what remains to be agreed upon, they have said enough to make it clear that substantial progress has been made. Since both believe that the strategic arms limitations talks are the linchpin of detente, there is every reason to expect a general improvement in relations.

But while we have to wait for official announcements to see precisely what has been achieved, the change in atmosphere is unmistakeable - and it is important, much as some people and disparage mere "atmospherics." The change would appear particularly striking to someone who left Washington a little while ago, as I did, when mutual recrimination and distrust were at their height, and returned to find that relations are on the mend. But an absence of this kind provides a good yard-stick with which to measure progress.

It is obvious that Gromyko's own return to the United States, albeit with a new set of proposals, also helped to convince him that circumstances were now move propitious. The intricacies of arms control may need careful negotiation, but Carter's attitude on human rights is almost as important to the Kremlin. Moscow used to argue that Carter's single-minded concentration on human rights was at cross-purposes with detente, which alone could make peace secure in a world full of nuclear weapons. It was an argument strongly supported by Kissinger.

Gromyko arrived in the United States to hear that "human rights cannot be the only goal of our foreign policy," that a choice that helps the United States to move toward one of its goals, such as human rights, "may move us further away from another goal." But this time it was not Kissinger speaking, but Carter himself, in an article written for the Baltimore Sun and widely reprinted throughout the world. This is the kind of change in atmosphere that has helped persuade the Kremlin that is is time to move on matters of substance.