President Carter yesterday demanded the release of all remaining American hostages in Iran and warned that any attempt to try them as spies "would be a further flagrant violation of elementary human rights, religious precepts and international law and practice."

In a strongly worded statement released by the White House, the president condemned threats by Iran's defacto ruler, Ayatollah Ruhollah Khomenini, to put the remaining American captives on trial if the United States refuses to hand over deposed Shah Mohammad Reza Pahlavi.

"The government of Iran is repsonsible for achieving their immediate and safe release, and the United States has the right to expect that Iran will do so," Carter said.

He added: "Worldwide outrage at the detention of the hostages would be greatly heightened by any attempt to put these diplomatic personnel on trial."

Carter's blunt warning came amid increasing hints that the United States is preparing some new initiatives to deal with the 16-day-old crisis that began Nov. 4 when militant Iranian students, egged on by Khomeini, stormed the U.S. Embassy in Tehran and took the occupants prisoner.

U.S. officials refused to disclose what these new moves might involve.

Instead, they said only that the immediate focus of U.S. efforts is to ensure that all the hostages being released under Khoneini's specially decreed dispensation for women and blacks first get safely out of Iran.

The first three hostages released are now in West Germany, and an additional 10 were flown out of Tehran this morning. That leaves an estimated 57 persons, 49 of them Americans, still captive and vulnerable to the threat of espionage trials mentioned by Khomeini an interviews Sunday with the three major American T.V. networks.

From the outset, lack of firsthand information about the situation inside the embassy has hampered an accurate count of how many hostages actually are being held. The State Department's original estimate of roughly 100 was scaled down to 70 late last week, and with 13 now released, the department's estimate is reduced to 57.

In response to questions about new U.S. moves, State Department spokesman Hodding Carter said only that the diplomatic channels through which Washington has been trying to negotiate with the Iranians are "very alive and very active."

"A great deal of work is going on throgh a number of channels," he said. "I think an answer will start to become clear soon. There should be some evidence in the next few days."

But while the spokesman and other U.S. officials refused to go beyond these cryptic remarks, there were increasing indications last night that one move might involve the return of the shah, who has been undergoing cancer treatment in New York, to the Mexican resort of Cuernavaca where he had been living until last month.

Despite warnings from Iran that the shah's departure to a third country would only make the situation worse, the Carter administration is believed to feel that getting him out of the United States would deprive Iran of a rationale for holding the hostages and increase its isolation in world opinion. r

A spokesman for New York Hospital-Cornell Medical Center, where the shah is a patient, said yesterday it will be "the end of the week or a few days longer" before the shah's doctors complete his present course of radiation treatment. Then, the spokesman added, the doctors want to attempt removal of a gallstone preventing further treatment with anticancer chemicals.

However, it was understood that tentative arrangements had been made for the shah to return to Mexico over the weekend or yesterday, but that the move had been postponed because of Iran's decision Saturday to release some hostages and Washington's desire to get as many out as possible before taking any steps to change the situation.

While refusing to discuss the specifics of any possible new moves, administration officials stressed the need for continued flexibility in the face of what they called changing and frequently contradictory signals coming out of Iran.

Iran, they noted, started out demanding the return of the shah, then added a demand that his personal wealth also be returned and, on Sunday, suddenly interjected the charges of spying by U.S. Embassy officials. As a result the officials said, the United States must be prepared to address sudden changes in the situation and cannot commit itself to any specific steps too far in advance.

The Treasury Department announced yesterday that it now estimates assets frozen by President Carter las t week at more than $8 billion -- a substantial jump from the administration's initial estimate of $5 billion. frozen assets is still under way and that the revised figures could change further.

President Carter spent yesterday at Camp David, where he is expected to remain through the weekend. But White House press secretary Jody Powell said Carter was keeping in close touch with the situation, and he underscored the president's statement on possible spy trials by saying:

"It is a violation of international law to place diplomatic personnel on trial. To hold them as hostages or place them on trial is a direct violation of international law, period. The question is not even legitimate."

Powell said the White House appreciated statements by leaders of black and women's groups criticizing Iran's decision to free only black and female hostages. He cited particularly a speech yesterday to the AFL-CIO convention here by Vernon Jordan, head of the National Urban League, who called the Iranian action " a cynical attempt to divide the American public. Black Americans refuse to be pawns in the Ayatollah Khomeini's insane games."