Secretary of State Edmund S. Muskie said yesterday America's relations with its European allies underwent "a difficult period" because of the Afghanistan crisis, but he added that the strains now have been replaced by "an atmosphere of mutual concern and support."

In testimony before the House Foreign Affairs Committee, Muskie said: "Let me be candid. We and our allies did undergo a difficult period several months ago. . . . The Soviet invasion of Afghanistan was a challenge to our common interests although not to our collective territory. As we searched for ways to make a coherent response, there were differences among us."

He also warned, "It will remain a central problem to prevent the Soviets from dividing our alliance -- and to head off the tempting but dangerous illusion that Europe can remain a stable island of detente while the Soviets create dangerous situations elsewhere. Dealing with this problem will require vigorous and subtle diplomacy . . ."

But, Muskie asserted, the problems and recriminations that were unleashed by efforts to work out a common strategy for confronting the Soviet threat have been eased greatly by such events as the recent Venice summit and the meeting in Ankara of North Atlantic Treaty Organization foreign ministers.

"The Soviet Union will make a great mistake to think that it can separate us from our allies on fundamental issues," Muskie said. "The results of Venice and Anakara make this point clear . . . Today the allies stand together on issues of interest that lie beyond the alliance area."

"The era of lockstep unity between the United States and its allies, to be sure, has passed," he conceded."The strengthened economic and political power of the other industrial nations, and their perception of their own interests, made this inevitable."

"But," he continued, to cast the current situation as 'disarray' or 'disunity' would be to misread profoundly the true state of our relations. The sum total of Venice and Ankara is mutual respect and cooperation."

Muskie, who was making his first public appearance before the committee since assuming office, stuck to that contention despite repeated questions from panel members about diminished American prestige in Europe and the failure of major U.S. allies to heed Washington's urging that they vote against a pro-Palestinian resolution in the United Nations General Assembly on Tuesday.

In response, Muskie said that the nine nations of the European Economic Community had abstained on the U.N. vote because they hope to play "a middleman role" in the Mideast conflict and thought that "an abstention was the best way to demonstrate their evenhandedness."

On the broader question of U.S. prestige, he noted that during recent weeks, he has met with 25 foreign ministers in Europe and the Far East and found all of them eager to solicit the views, support and advice of the United States.

"I was not made to feel like I was Little Orphan Annie in any one of these enviornments," Muskie said.

In regard to the Mideast, Muskie harshly criticized the U.N. resolution, which calls for unconditional Israeli withdrawal from occupied Arab territories, as "mischief-making" and "diversionary tactics" orchestrated by foes of the Camp David peace process. Asserting that "these tactics solve nothing," he said the best hope for a Mideast solution rests in pursuing the U.S. mediated talks between Israel and Egypt on Palestinian autonomy.

He also indirectly criticized the Israeli parliament for its action yesterday in passing legislation declaring Jerusalem the undivided capital of the Jewish state. He said, "All of the parties must avoid unilateral actions designed to prejudge the outcome of the negotiations or that would have the effect of worsening the atmosphere for successful negotiations."

However, Muskie added that the Carter administration has not yet decided whether it would instruct, the U.S. ambassador in Israel to refuse to meet with Prime Minister Menachem Begin in East Jerusalem if Begin goes ahead with a threat to transfer his office to that disputed part of the city.

In regard to the Iran crisis, Muskie said, "We continue to press, through every avenue open to us, for an early release" of the 52 American hostages there. While conceding that these efforts so far have not been promising, he said there are hopes that the outlook for resolving the hostages' plight will improve. He repeated that once a resolution is achieved, the United States would be willing to seek a new relationship with Iran including, if the Iranians wish, normal diplomatic ties.