One day after a senior Iranian official hinted at possible compromise on Iran's financial demands for release of the American hostages, a leading spokesman for hard-line Moslem fundamentalists again raised the prospect of placing the captives on trial should Washington "finally" reject Iran's demand.

The statement yesterday by Ayatollah Mohammed Beheshti, head of Iran's Supreme Court and leader of the dominant Islamic Republican Party, underscored growing differences within the Iranian ruling group on whether there is room for compromise regarding Washington's rejection of the multibillion-dollar demand for release of the 52 Americans held hostage since Nov. 4, 1979.

Beheshti spoke a day after Iran's top negotiator, Behzad Nabavi, said his government was "ready to listen" to any U.S. counterproposals to the Iranian terms.

A Tehran radio commentary broadcast in its domestic Persian service raised the possibility of execution of the captives, according to a transcript of the broadcast received by the State Department.

"According to Islamic laws, the trial of the hostages might, of course, not only lead to their execution, being guilty of spying, but even more importantly, it will lay bare the unwarranted interference of imperialism in other countries," the commentary said.

"The enemy should know that it has no other option but to give a positive response to Iranian demands and that procrastination might force Iran to adopt an uncompromising position, and consequently, the hostages might be tried."

It was unclear on whose authority Tehran radio had raised the threat of executions. The radio generally has reflected the views of the hard-liners, and its broadcast yesterday reaffirmed that "Iran is not ready for any softening of her position and will not deviate from" its demands.

State Department officials said last night that the execution threats, as well as the threats to place the hostages on trial, are a source of concern and that the United States is communicating its views to Iran through diplomatic channels.

The officials noted that this is perhaps the fourth time that a cycle of threats to place the hostages on trial has come from Iran but that threats of execution have been relatively rare.

A State Department spokesman repeated the standard U.S. position that the Iranian authorities will be held responsible for the safety of the hostages.

Washington officials were uncertain whether the Tehran broadcast represents the view of only that faction in Iran which currently operates the Tehran radio. They noted that while there has been recent mention of trials from some senior political figures in Iran, none of these figures has spoken of possible execution of the hostages.

There was some speculation in Washington that the broadcast may be an attempt to place additional pressure on the United States in connection with the response to Iran's demands, given to Algerian intermediaries on Tuesday, for transmission to Iran.

President Carter, on his return to Washington from Camp David yesterday, said the United States has made "reasonable proposals" for a settlement.

"I think it would be to the advantage of the Iranians -- certainly to the advantage of the United States and the hostages -- if they would accept what we have proposed," he said.

The United States has reformulated its response to Iran's terms, but has rejected a demand that it deposit $24 billion in Algeria to guarantee its acceptance of the four primary demands.

These demands, drawn up by the Majlis, Iran's parliament, are that the United States promise not to interfere in Iranian affairs, release Iranian assets, drop all legal claims against Iran and return the assets of the late shah.

Yesterday Beheshti linked rejection of the guarantee demand to placing the hostages on trial. "When the government announces that America has finally rejected the Majlis' conditions, then they should be tried," he said.

Beheshti was asked whether Iran could accept an arrangement whereby the hostages were released at the same time as an initial cash payment was turned over to Iran by the Americans.

"It should be discussed after they have done it," Beheshti said, indicating that Iran still wanted payment before the hostages' 14-month-long captivity could be ended.

A more flexible position was expressed Tuesday by Nabavi, the executive affairs minister who heads Iran's side of the hostage dealings. He said his country was open to U.S. counterproposals about the guarantees.

He also said alternative guarantees acceptable to Algeria would be accepted by Iran, indicating an enhanced but as yet unspecified role for the Algerians in the dispute.

In another development, Iranian Justice Minister Ebrahim Ahadi has resigned after less than two months in office. In a letter to Prime Minister Mohammed Ali Rajai quoted by Iranian newspapers, Ahadi cited "endless restrictions" on his authority as the reason for quitting.