Secretary of State Edmund S. Muskie said yesterday that the Carter administration is unable and unwilling to meet Iran's "unreasonable" conditions for return of the 52 American hostages, but that it will continue the diplomatic dialogue in search of a solution.

Muskie announced the U.S. position in unusually emphatic language a few hours after Iran published the text of its detailed demands for $24 billion in cash, gold and "guarantees" to be supplied by the United States. Administration policy deliberations continued yesterday on the Iranian proposals, which reached Washington on Friday.

Making public the U.S. stand, Muskie said Iran has proposed actions that are "beyond the power of the president to take" and beyond the established and unaltered guidelines of the administration in negotiating for the hostages' release.

The secretary of state made his comments in an interview on "Meet the Press" (NBC, WRC) after two telephone conversations yesterday morning with President Carter in which he summed up the findings of a meeting of high officials at the State Department.

Reflecting the pessimism in official circles since the receipt of Iran's proposals, Muskie said that it would be "very difficult" to bring about the release of the hostages through a negotiated settlement before the Reagan administration takes office Jan. 20.

Officials conceded privately that the indirect negotiations are close to a deadlock because of the wide gap between the United States and Iran, compounded by Iran's publication of what it insists is a "final answer" to American proposals.

The initial reaction of some Washington officials to Iran's detailed demands was to suspend the diplomatic dialogue and leave the matter for the Reagan administration. But this was never considered a likely course because such a position might cause Iranian authorities to place the hostages on trial. Muskie's statements on television and in talks with reporters after his appearance made it clear that the United States will seek to continue the diplomatic exchanges with Iran.

It remains undecided how and when the United States will proceed with an "official response" through diplomatic channels, as distinct from yesterday's "official reaction," as Muskie called his public words. State Department sources said the official U.S. response is likely to be delayed for a few days while an attempt is made to learn more of Iranian attitudes through conversations with the Algerian diplomats who have been acting as intermediaries.

Muskie suggested in his comments that Iranian authorities may shift their position, saying "this is not the first time they suggested items that would require us to go beyond the president's legal authority." Iran has modified its positions in several important respects over the 13 months since the hostages were taken.

And Muskie continued to express optimism about the final outcome, saying that "I think these hostages will eventually be released. I don't believe the Iranians can escape the costs that they are now paying for holding the hostages."

One official involved in the negotiations expressed the view that it would take six months to a year for a continuation of the current indirect talks to produce a settlement that brings about the hostages' release. But given the volatile climate of revolutionary and war-torn Iran and the unpredictable policy decisions of a new administration in Washington, sudden changes in direction or even crises seem at least as likely.

Both Muskie and Secretary of Defense Harold Brown, who appeared on "Face the Nation" (CBS, WDVM), declined repeated opportunities to suggest that U.S. military is under renewed consideration because of the difficulties in the diplomatic efforts. "We continue to intend to pursue peaceful approaches," Brown said.

As put forth by both officials, on the basis of conversations shortly before their television appearances, the administration is willing to restore "the status quo," with regard to Iran's assets, that existed at the time the hostages were taken -- but not to take tangible steps beyond that which could be seen as "ransom" for the hostages.

From this perspective, Iran's idea of monetary "guarantees" to be placed in Algerian hands is beyond the political bounds of the administration position. There is little doubt that the president could not establish such funds without congressional authorization, and there is virtually no chance that such action would be approved by Congress.

The newly released text shed new light on some points of the seven-week-old diplomatic dialogue, which began Nov. 2 with the decision by the Iranian parliament, or Majlis, to establish four conditions for the release of the hostages. The document disclosed:

Iran is asking the United States tacitly to acknowledge previous interference in its internal affairs by including the words "from now on" in a pledge not to interfere in the future. Neither side is treating this as an important problem, however.

Iran agrees to settle pending claims by U.S. citizens and corporations against its blocked assets "through arbitration acceptable to the respective parties." But the details of an arbitrating mechanism have not been spelled out.

Iran demands that $9 billion in bank accounts and funds, plus about $1 billion in gold, be deposited with Algeria before release of the hostages and released to Iran immediately upon release of the Americans. U.S. officials said that some of this, which is not encumbered by American claims, could be transferred to Iran when the hostages are released. But other assets, which are the subject of claims, could not be transferred to Iran until the claims are settled.

Iran is asking, in addition, for a $4 billion fund to be supplied by the United States to Algeria, evidently against the possibility that additional Iranian governmental assets will be found in this country. No such fund can or will be supplied under U.S. guidelines, officials said.

Iran is asking for a separate but similar fund of $10 billion to be placed in Algeria by the United States as a "guarantee" of transmittal to Iran of assets of the late shah, Mohammad Reza Pahlavi, and members of his family. Iran revealed that it had asked the United States on Dec. 10 for a list of shah-related assets, but has received "no useful information."

U.S. officials expressed doubt that any substantial part of the fortune of the late shah remains in this country, and that any large amounts held by members of his family are here.Even if they were, the United States could not transfer them to Iran without legal procedures that would take years, and there is no authority or inclination for the United States to transfer billions to Algerian hands against this possibility. the United States is not likely to make large changes in its previous positions in the next exchange with Iran. "We think we have offered them basically what we can," Muskie said.