President Carter's decision to raise a threat of military force for the first time in his presidency is an attempt to halt a drift toward espionage show trials for the American diplomatic personnel held hostage in Tehran, informed officials said last night.

The seemingly sudden reversal in the government's public posture toward military action raised the crisis to a new level of danger for all concerned. In the context of steadily mounting tension between Iran and the United States, it is certain to precipitate a new cycle of reaction and counterreaction and seems likely to hasten the 17-day-old confrontation to a climax.

The president's decision was made in early afternoon at Camp David, shortly before he flew back to the White House to supervise the preparation of a terse but weighty public statement.

According to informed sources, a combination of events relating to the possible spy trials convinced Carter that it was necessary to underscore publicly the consequences of harming the U.S. diplomats.

A strident new televised speech by Iran's leader, Ayatollah Ruhollah Khomeini, was the catalyst but not the only factor in the decision to raise a U.S. warning, the sources said. Khomeini had broached the posibility of spy trials in interviews with U.S. television networks last weekend. But his speech yesterday and perhaps other, undisclosed information added grave new concern.

There were many elements in the volatile and extremely uncertain situation that the United States faced last night:

Khomeini's increasingly strong suggestions that some or all of the 49 Americans remaining in the Terhan Embassy will be tried as spies. The State Department made it clear yesterda that such action would be even more repugnant than is the holding of the diplomatic personnel as hostages, it referred to possible trials as "illegal and unacceptable . . . a mockery."

khomeini's consistently contemptuous attitude toward the possibility of U.S. intervention. The Iranian leader said just after the embassy takeover that the United States is "incapable" of using force, "power-less" to do so. He continued in much the same vein yesterday. The unpredictability of his reaction now is heightened by an accident of timing: today's nationwide march in Iran to celebrate 1,500 years of the Islamic lunar calendar, potentially the occasion for high emotion.

Yesterday's invasion of Saudi Arabia's holiest mosque, the Great Mosque in Mecca, by armed men of uncertain origin. Some reports, still unverified last evening, suggested that followers of Khomeini's Shiite sect of Islam were responsible. Such a spreading of the Iranian crisis to the world's leading oil-exporting state would have enormous consequences.

The growing possibility of an early transfer of the deposed shah, Mohammad Reza Pahlavi, from his New York hospital bed to Mexico or elsewhere in the next few days after completion of this phase of his cancer treatment. Some Iranian militants have suggested that the shah's transfer from the United States could bring reprisals on the hostages rather than defuse the crisis. Therefore the specter of American military action could be a prelude to travel by the shah.

The uncertain attitude of the Soviet Union toward U.S. military intervention or the suggestion of it. Soviet President Leonid Brezhnev publicly warned the United States a year ago against any military action to shore up the falling shah, and the Soviets are extremely sensitive to U.S. forces on their border. On the other hand, the Soviets strongly oppose the taking of diplomats as hostages and have been quietly supportive of the U.S. tposture at the United Nations with regard to the crisis. Secretary of State Cyrus R. Vance met with Soviet Ambassador Anatoliy F. Dobrynin for 57 minutes yesterday morning largely to discuss Iran.

Carter's decision to raise the threat of military force was made with obvious reluctance. His public speeches have contained one sentence of which he was most proud -- and which, as he campaigns for reelection, has always brought applause from even his most critical audiences. It has been his statement that not one drop of American blood has been shed in combat during his presidency.

But from the time the Iranian militants seized the U.S. Embassy in Tehran with the encouragement of their government, administration officials realized that their last resort might have to be miliary force.

The last resort took on new urgency at 11:55 a.m. yesterday when a Reuter news agency bulletin from Tehran quoted Khomeini as announcing that the remaining hostages were spies and "would stand trial."

A meeting of the National Security Council's special coordinating committee, composed of the top foreign policy and defense officials, was already in progress at the White House. Two hours later, the official U.S. translation of Khomeini came across the government wires, showing that the Iranian leader's remarks were couched in somewhat more conditional -- though still menacing -- phrases.

The officials U.S. translation quoted Khomeini as saying: "If Carter does not send the shah, it is possible that the hostages may be tried, and if they are tried, Carter knows what will hapen."

The president, who had gone to his Maryland mountain resort at Camp David for the Thanksgiving week, was in touch with his White House adviesrs by telephone. They concluded, according to knowledgeable sources, that the ayatollah had so escalated the situation that the hostages could become "prisoners" after staged and hollow trials.

Carter made his decision by telephone that this was the time to raise publicly the military option. And he decided to return to Washington immediately to oversee the drafting of the U.S. statement and increase its resonance by his presence.

At 4 p.m. his helicopter landed on the White House south lawn, and Carter emerged, looking grim. Shortly after 5 p.m., White House press secretary Jody Powell issued the official U.S. statement that, in diplomatic nuance, threatened the use of military force if efforts at achieving a peaceful solution fail.

Carter returned to Camp David by helicopter to await the uncertain outcome of this latest escalation n the crisis in Iran.