Secretary of State Edmund S. Muskie cautioned yesterday that, despite the optimism being stirred by statements from Iran, "it would be a mistake to raise expectations" that the plight of the American hostages might be nearing an end.

Muskie's warning against premature optimism, made at a news conference here, appeared to go against the grain of President Carter's statement in Texas yesterday that recent developments in Iran "may very well" lead to a solution of the 10-month hostage crisis.

However, the secretary's cautionary note, which he stressed repeatedly, reflected what appears to be the assessment of those officials in the State Department and elsewhere within the administration who have been grappling with the Iranian crisis on a day-to-day basis.

Underlying their caution is the officials' feeling that, despite an apparent softening of Iranian demands signaled last week by Ayatollah Rudollah Khomeini, the intentions of Iran's revolutionary leaders remain unclear. And the United States, for all its probing, has been unable to open dialogue with Tehran that would clarify the Iranian position and make possible negotiations on the release of the hostages.

Muskie acknowledged that situation yesterday when he said: "We have a number of channels open to try and communicate messages of one kind or another. nOne never knows when one of those suggestions takes root and blossoms out in the kind of contact that results in negotiations. But that link has not yet been made."

In that connection, Muskie specifically denied as "inaccurate" a report by the Montreal newspaper La Presse that the State Department and former Iranian foreign minister Sadegh Ghotbzadeh had been engaged for three months in secret negotiations based on a document prepared by European intermediaries.

The La Presse report, Muskie said, appeared to originate with persons aware of one of the channels being used by Washington in an effort to establish contact with Tehran. But, he added, it had not lead to anything that could be described as actual negotiations.

Within the State Department, there is a feelilng, based on deduction, hunch and some cryptic signals from Iran that events there -- the installation of a prime minister and the decision of the new Iranian Parliament to address the hostage issue -- might be making time ripe for resolution of deadlock over 52 American captives.

That feeling was given new impetus when Khomeini, in a statement Friday, omitted from a list of conditions for freeing the hostages Iran's previous demand that the United States apologize for its past actions in that country. d

Similarly, Prime Minister Mohammed Ali Rajai, replying publicly last week to a letter from Muskie, talked in vague terms about American "repentance" but avoided giving any clear reaction of Muskie's call for "a channel of communication" between Washington and Tehran.

But, while these developments seem encouraging on the surface, U.S. officials have been cautioning privately for several days that hopes have been raised before about the hostages only to end in disappointment, that the situation is still too murky to make predictions and that the best course for the United States is to remain in a flexible, wait-and-see posture.

That was the message Muskie obviously was striving to get across yesterday, and he stuck to it in the face of repeated suggestions from reporters that he appeared to be at odds with the president's optimism. In response, Muskie said of Carter's Texas remarks:

"I have no reason to conclude he is hinting at anything. I think his statement is perfectly consistent with statements I've made that we have been working through as many channels of communication as we could find . . . with a view to ultimately establishing official contact that might lead to negotiations."

In a related development, the families of the 52 hostages revealed yesterday that they send a letter two weeks ago to the Iranian Parliament asking for a "face-to-face" meeting with Iranian officials that could become "the bridge that brings the hostages home." Muskie, who wore a yellow "free the hostages" emblem in his lapel, said in his news conference that he was aware of the letter.