The United States and the Soviet Union "lost an important opportunity for a mutually beneficial" agreement limiting nuclear missiles last year when both governments rejected an unauthorized proposal worked out by arms negotiators at Geneva, a new congressional report says.

Despite that rejection, "perhaps the best chance" for reaching agreement at the stalled talks on intermediate-range nuclear forces (INF) still lies in working out "something along the lines of that proposal," according to a staff report of the Republican-controlled Senate Foreign Relations Committee.

The proposal referred to is the one discussed last July in the now famous and controversial "walk in the woods" taken by chief U.S. negotiator Paul H. Nitze and his Soviet counterpart, Yuli A. Kvitsinsky.

Because there is still interest among U.S. allies in Europe in that Nitze-Kvitsinsky formula, support from the staff of a prestigious Senate committee could increase pressure on the Reagan administration to reconsider it in some fashion.

The informal proposal had considerable appeal in Europe, especially in West Germany, because it would have limited the scope of rival missile deployments in Europe and put a cap on Soviet missile deployments in Asia. It would have allowed 300 single-warhead, jet-powered cruise missiles on the U.S. side and 75 triple-warhead SS20 missiles for the Soviets in Europe. But it also called for dropping the planned deployment of 108 U.S. Pershing II missiles in West Germany.

The Pentagon balked at this, arguing that the rocket-powered Pershing was needed to balance the SS20 because the slow-flying cruise missiles take hours to reach their targets and can be shot down by air defenses. It is not clear why Moscow also repudiated the deal.

President Reagan, at a press conference last week, appeared once again to rule out any such proposal. But there were reports Saturday from Geneva, where West German Chancellor Helmut Kohl visited Nitze, that German interest in some such arrangement continues.

The staff report, released today, paints a gloomy picture of U.S.-Soviet relations and says there is "little chance of agreement in the near future" on arms control. Prospects in the strategic arms reduction talks (START) of dealing with intercontinental-range missiles are even "worse" than the INF talks dealing with shorter-range missiles, it says. Both talks are in Geneva.

"Perhaps the most disturbing finding," the report says, "is the extent to which the current arms control impasse has led to a deterioration" in the overall U.S.-Soviet relationship. Any improvement, it says, also "will depend to a large extent on whether agreements can be reached on limiting nuclear weapons."

The five senior staff aides who wrote the report after a March trip to Moscow and other European capitals, said Soviet officials described relations with the Reagan administration as "a critical situation . . . pushing us to the brink" and "a highly dangerous path" that they view with "extreme pessimism."

At the same time, the staff aides report that Soviet INF negotiators were "belligerent" and "nasty as hell" during the last round of talks early this year, according to U.S. officials, in an effort to ensure that there was no sign any progress was being made.

The Soviets want to influence European public opinion against plans to install new U.S. missiles in Europe beginning in December if no arms accord is reached before then. Western officials say they believe that Moscow is trying to block deployment without an agreement and without giving up its own modern missiles already fielded.

The staff report, according to committee Chairman Sen. Charles H. Percy (R-Ill.), is meant to set the stage for congressional hearings on U.S.-Soviet relations to begin June 15 with Secretary of State George P. Shultz as the lead-off witness.

The staff aides' March visit to Moscow came shortly after Reagan speeches describing the Soviet Union as an "evil empire" and calling for a space-age defense against Russian missiles. Thus, the report states, tensions appeared to be especially high although "it is not always possible to know what is said for effect and what is genuinely believed" by Soviet officials.

More recently, however, Reagan has won victories in Congress on the MX missile and pledges at the Williamsburg summit of alliance unity on arms. The president is at Camp David this weekend studying changes in the U.S. START proposal recommended by a special bipartisan commission on strategic forces.

White House aides feel all of these developments have strengthened the president's hand and may improve chances for progress in negotiations with Moscow. There were also signs from Soviet leader Yuri V. Andropov, in discussions with former U.S. ambassador W. Averell Harriman last week, that Moscow, too, may be interested in improving relations.

But debate continues in the administration over the wisdom of a summit meeting between the two leaders, with the stalemate over arms control a major stumbling block.

Some officials are said to feel that it may be possible to set aside the absence of a final agreement on arms. They add that the kind of agreement in principle on arms that then-President Ford and Soviet President Leonid I. Brezhnev achieved in Vladivostok in 1974 might be duplicated in a Reagan- Andropov meeting. Such a step then could pave the way for an accord and make progress possible on other fronts.

Last week, Reagan also forecast that U.S.-Soviet relations would improve in the long run but that there would undoubtedly be more hostile rhetoric as the showdown approached over the planned U.S. missile deployments in Europe.

Thus the staff report, praising the squelched Nitze-Kvitsinsky deal, is seen as potentially important. West Germany, which is the only country slated to receive the Pershing missiles, has been singled out as a target for pressure by Moscow.

The report said that as public demonstrations in Germany increase in intensity later this year it is clear that Bonn will press Washington for additional modifications in its proposals. Some U.S. officials expect Moscow to put forward a new proposal at the talks this fall which would be sufficiently appealing to cause European opponents of deployment to call for a moratorium on the scheduled December installation of U.S. missiles while further negotiations are held.

The congressional trip also came after disclosure of a memo from aides to chief U.S. START negotiator Edward L. Rowny. The memo included disparaging remarks about four of Rowny's five deputies. The Senate team visited the START delegation and reported that "Rowny's subordinates maintained their professionalism and tried not to allow the incident to interfere with the immediate business at hand.

"Behind the scenes, however, morale on the START team has deteriorated to the point where its future effectiveness could be seriously impaired," the report says.

Two deputies have since left the team, although the relationship of their departure to the memo is not clear. Rowny's top deputy, Ambassador James Goodby, is also known to have been upset by the memo but is understood to have accepted the apology Rowny offered to the team. Goodby will return with the delegation to Geneva on June 8 for the next round of talks. But it is known that he is looking for a new job.

The report said U.S.-Soviet talks in Vienna dealing with mutual and balanced force reductions (MBFR) is the one place where there are some prospects for an accord. The staff calls for "serious probing of the East's intentions" in the round beginning there this month.

The staff also compiled the views of unnamed "Soviet officials and strategic analysts" which suggest that they are not too different from the views of the recent special U.S. commission on strategic forces.

The Soviets said the problem of land-based missiles becoming vulnerable to attack by enemy land-based missiles "is greatly exaggerated, but it could become a problem" if the U.S. missile buildup goes ahead. The United States says the same about the Soviet missile threat.

The United States, the Russians are quoted as saying, should not fear an attack on its missiles because it would have to be so big that it would be seen as an attack on the entire country and demand retaliation.

The U.S. buildup of more accurate missiles eventually will "force us out of our silos," the Soviets are quoted as saying, meaning they may also move to mobile missiles but this will create problems of verifying any arms agreement.

The Soviets also were quoted as saying that moving from multiple-warhead to single-warhead missiles, as the U.S. commission recommended, "is a good idea but will take a lot of time and money."