Soviet-American relations have failed to improve in an initial round of mutual probing by a new leadership in the Kremlin and the second-term Reagan administration, in the view of senior Soviet officials and analysts.

That probing now appears to have reached a critical stage, where decisions by both sides could set the course of the U.S.-Soviet relationship for the rest of the Reagan administration, in the view of these influential Soviet figures.

Any expectations that President Reagan's sweeping reelection victory and the accession to power here of Mikhail Gorbachev, the 54-year-old, dynamic new Soviet leader, might open the way for a dramatic bid to improve relations have completely evaporated now, these Soviets suggest.

If anything, the tone of the relationship has become more strident in recent months as both sides appeared at first to move toward a get-acquainted meeting of their leaders and then backed away in mutual suspicion that the other side was laying a propaganda trap.

Even small gestures that might have been expected to reduce tensions seem to have gone awry. A visit to Moscow last month by Secretary of Commerce Malcolm Baldrige, whom Gorbachev received, produced no visible results on trade issues and has left a bad taste in the Soviet capital.

And confidence-building measures proposed by Reagan in his May 8 speech at Strasbourg, France, such as establishing a hot line between U.S. and Soviet military commanders, are dismissed as insincere or meaningless by the Soviets, who focus instead on what they describe as the president's apparently willful failure to mention the Soviet Union's role in World War II in that speech.

"We might welcome some of the steps the president mentioned in the context of a certain policy," said Georgi Arbatov, head of the Soviet Union's Institute on the United States and Canada. "But that is not the case. Even if you have 10 hot lines in a dangerous situation, it still would not be productive. It is the policy that is the problem.

"It would have been better in the speech on the 40th anniversary of the end of World War II to have mentioned the Soviet Union than to propose these things."

Arbatov is a member of the Communist Party's Central Committee and his institute is believed by U.S. specialists to be the source of much official Soviet thinking on arms control and on relations with the United States.

An underlying theme in conversations with Soviet foreign policy advisers at this point is that the Soviet Union to a large extent already has disengaged from trying to formulate policies based on Reagan administration actions or proposals.

Unstated but evident in many of their comments is the idea that the Gorbachev team, after an initial assessment, is close to deciding to wait out the next three years, seek public opinion gains as no real business can be done now and see what the next U.S. administration offers.

"It is not important how we appreciate an action taken by President Reagan," said Vladimir B. Lomeiko, director of the press department at the Foreign Ministry and a close associate of Foreign Minister Andrei Gromyko.

"It is not important what his own attitude toward the antiballistic missile treaty, or toward the launching of new submarines, is. What is important is that he and his administration must get used to the idea that it is necessary to stop stockpiling nuclear weapons now. What is important is the political and philosophical approach. A patronizing attitude by one side to the other will not bring progress."

Lomeiko, speaking before Reagan announced his decision on accepting the SALT II accord's limit of 1,200 multiple nuclear warhead launchers, appeared to be discounting in advance the impact of Reagan's decision by stressing the independent nature of Soviet policy now.

Lomeiko reacted testily when asked about a recent news agency report quoting Pravda editor Victor Afanasyev as having said that Gorbachev was likely to visit the United Nations in the fall. This report, which associates of Afanasyev now say distorted his remarks, stirred speculation that Gorbachev and Reagan might meet in New York, an idea both sides now appear to have rejected.

"It was not we who put forward this idea" of a Gorbachev trip to the United Nations, Lomeiko said.

Asked about the possibility of a meeting between the two leaders in another locale within the next year, Lomeiko emphasized the problem in arranging such an encounter.

"It is not a simple process," he said. "First and foremost, it has to be carefuly prepared. It is an issue that depends on the two sides."

His reply did not openly contradict the generally positive and unspecific response that the Russians have given to Reagan's suggestion that a meeting take place. But it did appear to echo sentiments being expressed in Moscow now that the main objective of each side is to avoid being blamed for the appearance of deadlock.

The continuing bitter dispute over arms control has shifted from Soviet pressure to block the deployment of U.S. missiles in Western Europe to intense Soviet opposition to Reagan's Strategic Defense Initiative for a missile defense in space. It appears to have persuaded the Soviets that nothing more than symbolic gains can be made at a summit meeting with Reagan at this point.

Moreover, the Soviets appear to have concluded that the Reagan administration has reached the same belief and is maneuvering for maximum advantage in the game of public relations by insisting on Gorbachev's coming to Washington as the price for a summit meeting.

Both Soviet and American officials are extremely reticent in discussing the meeting between Gorbachev and Baldrige, and few details seem to have filtered out. But some reports circulating here suggest that the Soviets concluded that Baldrige had been engaged in testing Gorbachev. What may have been an acrimonious exchange probably has retarded chances for an early summit.

The Gorbachev-Baldrige meeting "was not a watershed in Soviet-American relations," Arbatov said. "It did not make any improvement . . . . In fact, we continue to be in something close to a state of economic warfare."

Gorbachev's decision to see the U.S. Cabinet member probably was intended "to show that he honestly is for an improvement in Soviet-American relations," according to Arbatov. "He had to deal with the sheer fact of how his saying no to a request from Baldrige to see him would be assessed outside the Soviet Union."

Guarded comments by Soviet officials suggest they would prefer a summit site with political symbolism that favors their causes, such as the Finnish capital of Helsinki, where the Soviets in 1975 received recognition of the post-World War II borders in Europe in return for promises on human rights.

Reagan, in July 1975, was sharply critical of then-president Gerald Ford's acceptance of the Helsinki accords, which were signed the next month, and his going to Finland would probably present his advisers with the kind of problems that Washington seems to represent for Gorbachev's aides.

Both Arbatov and Radomir Bogdanov, his principal deputy, said in separate conversations that the West's failure to respond to two proposals had particularly dimmed hope for progress. They were a moratorium on medium-range missile deployment in Europe announced by Gorbachev, and his reiterating a Soviet negotiating proposal to reduce the number of SS20 missiles targeted on Europe to the combined level of French and British nuclear forces.

"There was some misinterpretation of what the general secretary had proposed in announcing the moratorium," which is due to expire in November, Arbatov said. "It was said that we were trying to freeze an advantage into place.

"That is not the case. So it was necessary for him to repeat that we stand by our proposal to negotiate" down to the French-British levels "and to seek deep reductions, 25 percent or more," in strategic missile forces.

"There is a growing understanding that it is important to have the support of the public, and to have our positions clearly understood," Arbatov said. "That is a more important political factor than ever before, even if it is not always as well understood as I would wish."