Secretary of State Edmund Muskie and Soviet Foreign Minister Andrei Gromyko left a three-hour meeting here today with no visible sign of progress toward resolving the deep and increasingly bitter differences between the two superpowers.

Muskie said after the talks that the session fully justified his belief that such a "long and serious" session was "necessary." But he said further discussions "at some point along the way," apparently not scheduled now, would be necessary to produce "a resolution of the differences which exist."

The secretary of state, looking tired after his lengthy meeting with Gromyko -- which included only their interpreters -- said he would report to President Carter before saying more.

Gromyko, who usually has appeared side-by-side with U.S. secretaries of state after past talks, did not appear at such a post-meeting news conference.

Later, Gromyko did tell reporters of having "discussed international questions, including some important ones. Obviously, on all issues we differed." He added: "Still, we must say that it is essential to work for ways of normalizing relations. . . . We must find a way to conduct the course of detente and work for peace, not for tension."

The session, which ran an hour longer than expected, was the first at this level since the Soviet invasion of Afghanistan. It took place in the same Hofburg palace where President Carter and Soviet President Leonid Brezhnev signed SALT II 11 months ago, proclaiming their hopes for growing friendship and cooperation.

The occasion for the presence of both foreign ministers was the 25th anniversary celebration of a previous agreement between East and West, the establishment of neutrality in this border nation, Austria.

Neither friendship nor any sign of agreement was evident today. The much-anticipated conference was preceded by an open display of discord at a public ceremony.

Muskie had made no secret of his intention to present Gromyko in their private session with an unvarnished version of the U.S. attitude toward the Soviet invasion of Afghanistan, which the secretary has described as unprecedented and cause for a "sea change" in U.S. public and political beliefs about Soviet intentions in the world.

Muskie also said publicly in recent days that the fate of the second strategic arms limitation treaty as well as a broad range of Soviet-U.S. cooperative programs depends on Soviet withdrawal from Afghanistan. But there is no sign that Moscow is prepared for an early pullout.

Neither side had raised much hope that the meeting would produce tangible steps toward closing the widening gulf between them over Afghanistan, the NATO military buildup in Europe the boycott of the Moscow Olympics and a lengthening list of other questions. But both Soviet and U.S. officials welcomed the chance to reopen the top-level dialogue, and Muskie seemed to share that view when the meeting was over.

According to a description provided by a senior U.S. official, the meeting was not acrimonious but neither was there any report that it was productive in resolving U.S.-Soviet differences.

Gromyko reportedly told the new secretary of state at lunch, "We've been trying to find out who Muskie is. We hear some stories that he's tough and hard and had a terrible temper and some that he's calm and reasonable." Muskie said he responded, "I hope you never resolve that doubt."

The two had met briefly when Muskie visited Moscow in 1971.

Muskie's taciturn report tonight, which lasted barely a minute, was in sharp contrast with his outspokenness on nearly every previous occasion since he succeeded Cyrus Vance as secretary of state a little more than a week ago.

Prior to the Gromyko meeting, Muskie met with the foreign ministers of Britain, France and West Germany to discuss Western strategy. State Department spokesman Hodding Carter III said "no common front" had been forged among them about how to respond to the latest Soviet-endorsed hints of a future withdrawal from Afghanistan after all resistance ceases.

"Every government will study the proposal and probe it on its own," Carter said. Each of the other Western foreign ministers in this group is scheduled to have a separate session with Gromyko today or tomorrow.

Muskie also worked to shore up European support for a boycott of the Moscow Olympics in a separate meeting with French Foreign Minister Jean Francois-Poncet early today. The U.S. government has been strongly critical of the French Olympic Committee's decision to go to the Moscow Games and has not given up hope that the French government will intervene somehow to stop this from happening.

In the face of continuing reports that European nations are wavering in their support of U.S.-sponsored economic sanctions against Iran, State Department officials refused to make predictions about a forthcoming decision on the matter this weekend by the European Common Market.

It was clear that Muskie had failed to disuade European countries from introducing a plan of their own for a solution of the difficulties in the Middle East. The allies were reported still convinced that they have a role to play in making new suggestions, but they are said to be undecided what action they will take.

The presence of Muskie and Gromyko under present circumstances turned the commemoration of the 1955 East-West agreement on Austria into a search for lessons for the present from that achievement.

Austrian Chancellor Bruno Kreisky, an outspoken statesman whose country is wedged between the communist and capitalist worlds, opened today's formal ceremony by declaring that neutral Austria is "a visible example that coexistence does work, that peace is possible in our time."

Sitting in the front row on gilded chairs of the former royal palace were the foreign ministers of the four former occupying powers, whose predecessors in office had negotiated the 1955 agreement.

Gromyko, the senior foreign minister present with 23 years in his job, responded first. He praised Austria as a successful example of the fruits of "peaceful coexistence" and declared that the Soviet Union continues to support East-West detente, although he said without elaboration that it "has been tested" in recent months.

French Foreign Minister Francois-Poncet them made the first clear reference to Afghanistan, saying that Austria is "a nation restored" after "a millitary occupation that was not justified in any way." Today, more than ever, "understanding and not force is necessary,' he added.

British Foreign Secretary Lord Carrington spoke of the "dangerous times" of the present, in which tensions and modern arsenals are growing, and called the Austrian treaty "a timely reminder of the fruits which firm and patient negotiations can bring."

It was left to Muskie, the junior in point of service with only eight days in the job, to speak directly of the crises of the moment. In the most openly political of the speeches, he nevertheless managed to do so without actually mentioning either the Soviet Union or Afghanistan by name.

"The principles of neutrality, of independence and territorial integrity, so respected in the case of Austrian are today being violated," Muskie declared. The lesson is "that an act of aggression anywhere threatens security everywhere" and the consequence is that "we" -- apparently meaning the Western alliance as well as the United States -- "shall continue to convey the costs of aggression so long as it continues."

Gromyko, from his front-row seat, looked uncomfortable and yet disdainful, as if he were smelling or seeing something unpleasant. But at the end of Muskie's address, Gromyko joined in polite applause.

Following the speeches, the four foreign ministers went off to a ceremonial lunch, hosted by the Austrians, before Muskie and Gromyko sat down for their head-to-head meeting.