The State Department yesterday announced the next step in the diplomatic effort to free the 52 Americans held in Iran as President Carter and President-elect Ronald Reagan publicly expressed frustration with the lengthy hostage-keeping.

Carter, on a walking tour of Plains, Ga., said the prospects for "early release, I think, are unfortunately quite dim." Reagan, in Los Angeles, called the Iranian captors "nothing better than criminals and kidnapers who have violated international law totally."

The State Department announcement said the three officially designated Algerian intermediaries will fly here this weekend to discuss the official U.S. response to Iran's demands for $24 billion in asets and "guarantees" in return for freeing the hostages. The Carter administration has unofficially rejected the demands, calling them "unreasonable" and beyond the powers of the president to meet.

Administration officals said they expect to present the Algerians with a preview of the proposed official response to Iran. There is a substantial possibility, sources said, that the Algerians will leave Washington early next week carrying the U.S. reaction to be conveyed to Tehran.

About the best that can be hoped for, according to offical sources, is that Iranian authorities will decide to continue the dialogue by sending back another message, leading sooner or later to a modification of their position in a manner acceptable to the United States. Even in this best case, the Iranians are unlikely to do much until they see the posture of the Reagan administration, the sources said.

On the other hand, there is at least as much likelihood that Iran's demand for payments and guarantees really is the "final answer," as billed in Tehran. In this case, administration officials said, the Iranian authorities may terminate the indirect negotiations with the United States or even place the hostages on trial.

There is no sign that the incoming Reagan administration is close to a decision on how to handle the problem, which it is now nearly certain to inherit on Jan. 20. While there is a growing belief that the Carter administration's attempt at reaching a negotiated settlement is close to failure, the alternative to its line of policy is not in view.

Reagan's expression of anger and indignation yesterday did not suggest what he would do. "I think all of us down deep inside have an anger . . . at the idea that [the hostages'] captors today are still making demands on us for their return, when their captors are nothing better than criminals and kidnapers who have violated international law totally in taking these innocent people and holding them this long," Reagan said.

State Department officials as well as sources in the Reagan camp said there is no plan for members of the Reagan entourage to sit in on the impending discussions with the Algerian emissaries. But a Reagan source said this has not been ruled out.

Secretary of State Edmund S. Muskie, speaking to reporters in a breakfast meeting, rejected suggestions in newspaper editorials that the United States terminate the indirect negotiations.

To take this course, Muskie said, "might precipitate the trials" of the hostages -- which Iran has threatened and the United States has opposed in the strongest terms.

The United States has hinted in the past that it might take military action if any of the Americans are placed on trial. Asked about the trials issue, Carter said yesterday that "more than a year ago we let the Iranians know the consequences of any trial and I don't think it is necessary for me to repeat that."

Describing the course that he prefers and that the Carter administration has adopted, Muskie said that "we must continue our efforts, playing it steady, keeping the channels open. Negotiations are an evolutionary process . . . a process of persuasion and counter-persuasion."

Muskie recalled that for many months the United States sought in vain to establish a clear line of contact with Iran to discusss the hostage issue. "To cut it off ourselves and once again create the helpless feeling of floating and drifting, to abandon ourselves to whatever moods there are in Tehran, doesn't appeal to me," he said.

Muskie, in his meeting with reporters, said the recent Iranian demands were "a hard-line turn" in Tehran's position that came with "no warning whatsoever" to the United States. The money demands "have all the appearances of being a last-minute add-on to their position, stimulated by political differences in Tehran," he added.

The developing situation seemed favorable right up to the presentation of the lengthy and detailed Iranian demand late last week for $24 billion in payments and guarantees, Muskie said.

"We were proceeding in the direction of a claims settlement commission" to settle U.S. private claims against Iranian assets in this country. "There were some exchanges in the direction of clarifying that concept, of exploring its utility.

"As we approached the time of their reply, there wre a series of statements by Iranian leaders saying we were close to agreement. A sense of optimism was generated. The atmosphere was one of movement. We assumed that this movement would be within the limitations [on U.S. action] that we have spent a couple of months making clear to them," Muskie said.

He said it appeared to him that some of those who influenced Iran's decision "refused to understand the limits" of the U.S. stand.He placed major emphasis, beyond this, on a reemergence in Iran of an internal struggle for power, with Ayatollah Ruhollah Khomeini, the supreme authority, contributing to the problem by playing off one opposing group against another.

Muskie said that shortly before Iran's decision to seek the $24 billion, President Abol Hassan Bani-Sadr, who has been considered a moderate, was quoted as saying that the clerical factions running the prime minister's office were "giving away" Iran's position on the hostage issue. Muskie speculated that this stiffened the Iranian demands at the last minute.