Secretary of State Edmund S. Muskie said yesterday that the Soviet Union has modified its opposition to negotiating limits on medium-range missiles in Europe in ways that merit "constructive" study by the United States and its NATO allies.

Muskie gave this hint of possible movement toward new East-West arms talks after West German Foreign Minister Hans-Dietrich Genscher briefed him and President Carter on the just-completed Moscow talks between Chancellor Helmut Schmidt and Soviet President Leonid I. Brezhnev.

Muskie declined to discuss the latest Soviet reaction to Western proposals for limits on the so-called Theater Nuclear Force missiles, saying that Schmidt will lay out the details when he addresses the West German parliament, the Bundestag, in Bonn today.

But, Muskie said, the Soviet position outlined to Schmidt in his Moscow talks Monday and Tuesday was different from the flat rejection the Soviets gave the NATO proposals in January.

As a result, he added, he and Carter had agreed with Genscher that the NATO partners "will study this reaction in a constructive spirit -- that they are worthy of that kind of consideration."

Asked whether that meant Schmidt had "hit paydirt" on his controversial trip to Moscow, Muskie cautioned: "I think that would be a premature conclusion until the United States and our allies have had a chance to study the reaction."

Genscher, who appeared before reporters with Muskie after the White House meeting, turned aside similar questions by saying: "You make your own assessment tomorrow after you have heard the speech of the Chancelor."

Still, Muskie's comments appeared even more upbeat than the deliberately vague hints of progress given by Schmidt on Tuesday at the end of his Moscow trip -- a trip that had caused strains in U.S.-West German ties because of American fears that it would jeopardize NATO plans for deploying new missiles in Europe and lessen the pressurue for Soviet withdrawal from Afghanistan.

These U.S. misgiving, which are known to have been of particular concern to White House national security adviser Zbigniew Brezezinski, led to Carter's sending a brusque letter to Schmidt two weeks ago warning against any attempt to modiify the NATO missile decision.

Schmidt reacted with considerable anger, and a special meeting between the two leaders during the recent Venice economic summit was requiried to smooth over the matter.

In a tactful effort to keep the renewed harmony intact, Muskie recalled that Carter, following his talks with Schmidt in Venice, had endorsed the chancellor's Moscow trip.

Muskie added that, after hearing Genscher's report on the Moscow summit, Carter feels that Schmidt had conveyed to Brezhnev the position of the western alliance on both Afghanaistan and missile deployment "with firmness and clarity."

Muskie confirmed, as had Schmidt earlier, that the Moscow talks gave no sign of Soviet willingness to pull their occupying forces out of Afghanistan. sWhether a different story is in the offing on the missile issue will become clearer today after Genscher returns to Bonn from his briefing mission here and Schmidt breaks his silence on what he was told by Brezhnev.

At a ministerial meeting in Brussels last Dec. 12, NATO agreed to deploy U.S. Tomahawk and new-generation Pershing missiles in West European bases, from where they would be able to strike deep inside the Soviet Union. These weapons, whose deployment is to begin in 1983, are intended to balance new Soviet 2,500-mile-range SS20 mobile missiles already in place and scores of new Soviet Backfire bombers.

The decision was extremely sensitive politically in a number of NATO countries in part because of a heavy Soviet campaign of conciliatory gestures and threats designed to put pressure on the European allies not to go along.

To relieve the pressure, NATO coupled its missile-deployment decision with an offer to open talks between the United States and the Soviet Union on limiting their use.

The NATO plan, conveyed to the Soviets by Muskie's predecessor, Cyrus R. Vance, said limits on U.S. medium-range missiles should be accompanied by similar Soviet limitations, that the negotiations should go between Washington and Moscow within the context of the strategic arms limitation treaty (SALT) talks and that their objective should be to limit land-based missile systems in Western and Eastern Europe.

On Jan. 3, the Soviets, through their embassy here, officially rejected the NATO offer and said there could be no limitation talks unless NATO publicly abandoned its plan to deploy the Tomahawks and Pershings in Western Europe.

Part of the U.S. concern about Schmidt's trip was caused by fear that he would try to break the impasse by proposing a mutual three-year freeze on new deployments. Schmidt has denied that was his intention, and he said Tuesday the Soviets, in any case, are not interested in such a pause.