The Carter administration said yesterday it will counter Iran's demand for $24 billion in financial guarantees by telling the Iranians the United States will not change its basic proposals for freeing the 52 American hostages, but the administration appeared to be having difficulty in deciding how to get the message across to Tehran.

For the third straight day, U.S. officals led by Secretary of State Edmund S. Muskie met with Algerian intermediaries in an effort to frame a response to the latest Iranian demands. But after a marathon session that lasted into the night, the talks were adjourned until today.

State Department officials said they hoped the U.S. reply can be hammered out today. But, they added, they could not say with certainty when the United States will be ready to make its next move in the two-month parrying over the hostages.

The one thing that seemed clear was that the United States, which on Sunday made public the details of what it is willing to do to meet Iran's conditions, feels it has come close to the outer limits of the concessions it can make. As State Department spokesman John Trattner said:

"We are exactly where we can be. The Iranians will have to understand what we can do and what we cannot do."

His comments suggested that any new U.S. message to Iran essentially will stress that the administration is sticking with the proposals it made in mid-November and that were reformulated and elaborated on later.

However, although there now seems no hope of breaking the deadlock before President Carter leaves office Jan. 20, Trattner and other department officials said the administration wants to keep the indirect negotiating channel with Tehran open so that President-elect Ronald Reagan will have the option of picking up the process when he takes office. "We will not abandon the process," Trattner said.

Details of the talks with the Algerian intermediaries -- Redha Malek, the ambassador to Washington, and Abdel Karim Gheraieb, the ambassador to Tehran -- have been kept closely guarded within a small circle of senior administration officials. But, U.S. sources hinted, the delays in agreeing on a new American message appear to stem largely from questions on how it should be phrased.

According to these sources, the United States wants to convey to Tehran the unmistakable impression that it is standing firm on its earlier proposals and considers the Iranian demands incompatible with U.S. law and national honor. But, the sources added, Washington wants to get these points across in a way that avoids confrontation and keeps alive the possibility of further negotiation and compromise.

The impasse stems from Iran's insistence that all financial issues between the two countries be settled -- and that the United States guarantee Iran's stake by depositing $24 billion with the Algerian Central Bank -- before the hostages can be released.

As the documents made public by the State Department Sunday indicate, the United States has agreed to go a long way toward meeting the original conditions set by Iran's parliament on Nov. 2. The U.S. offer would include the immediate transfer of $2.5 billion in blocked Iranian funds, binding arbitration by an international commission of private American legal claims against Iran, and a freeze on the assets in this country of the late shah and his family.

To carry out these proposals, the administration even had prepared a series of presidential directives to be issued by Carter at the time of the hostages' release. But the Iranians, although indicating that they accepted the U.S. proposals in principle, then demanded that the United States provide what amounts to a $24 billion good-faith guarantee.

That, U.S. officials have said, involves concessions that are beyond the power of any American president, and it provoked outrage in this country, including harsh statements by Reagan that Iran is holding the hostages for ransom. Although the administration has been circumspect in its public pronouncements, Trattner said yesterday that "the word of the president of the United States is the highest authority in this country" and must be accepted as such by Iran.

Referring to Gheraieb's meeting with the hostages in Tehran over Christmas, Trattner said the Algerian envoy, while telling U.S. officials it was his "general impression" that the captives were in good physical condition, also had stressed that he is not a physician and could not testify with certainty to their physical and psychological state.

The spokesman also asserted that film made of most of the hostages at Christmas had not allayed official U.S. concern that some have been confined in prison and have had inadequate medical care -- charges the State Department made last week after Iranian officials said the captives were being kept in luxury hotels.

"Those concerns continue to be valid until we have the sort of regular, reliable information to prove to the contrary," Trattner said. "We would like to have them seen regularly by objective, qualified outside observers such as the Algerians."