Cyrus R. Vance, saying he had been unable to support President Carter's "difficult decision" on the unsuccessful attempt to rescue the American hostages in Iran, formally resigned as secretary of state yesterday.

Appearing before reporters at the State Department, Vance made public a letter of resignation he had sent to the president on April 21 -- four days before the aborted mission that left eight American servicemen dead.

White House sources said last night that Carter will move quickly to name a successor, possibly as soon as a presidential news conference scheduled for 9 o'clock tonight. Although the president reportedly had not made a decision by late yesterday, most informed speculation tended toward the idea that he will choose acting secretary of state Warren M. Christopher.

Other names being mentioned most frequently included Sol M. Linowitz, Carter's special Mideast negotiator; national security affairs adviser Zbigniew Brzezinski and White House counsel Lloyd Cutler. However, the dominant opinion in foreign policy circles seemed to be that they are less likely than Christopher to get the nod.

In his letter, Vance said he was resigning "with a heavy heart," and added: "I know how deeply you have pondered your decision on Iran. I wish I could support you in it. But for the reasons we have discussed, I cannot."

He continued, "You would not be well served in the coming weeks and months by a secretary of state who could not offer you the public backing you need on an issue and decision of such extraordinary importance -- no matter how firm I remain in my support on other issues, as I do, or how loyal I am to you as our leader."

In a handwritten letter of reply dated yesterday, Carter accepted the resignation "with regret" and told Vance: "Because you could not support my decision regarding the rescue in Iran, you have made the correct decision to resign. I know this is a matter of principle with you, and I respect the reasons you have expressed to me."

The resignation, which Vance made clear would have stood even if the rescue attempt had succeeded, cast a cloud of confusion and uncertainty over the immediate course of U.S. foreign policy, and everyone involved sought to reassure the public and America's allies that steps were being taken to keep the policy wheels turning in an orderly and consistent fashion.

Attempting to damp down speculation about possible further military moves against Iran, White House press secretary Jody Powell denied that Vance had lost a power struggle pitting his advocacy of moderate diplomacy in dealing with the Iran crisis against the tougher line allegedly backed by Brzezinski, the other major architect of administration foreign policy.

"It was an honorable difference over principles," Powell said about Carter and Vance. "Both men regret it, but the decision was unavoidable."

Vance similarly moved to smooth over the jolting effects of his action by stressing his support of the president in all other foreign policy matters and by calling on his loyalists within the State Department to stay at their posts.

His appeal, which he reinforced both in personal talks with his key lieutenants and in a meeting with the department's assistant secretaries, appeared to have halted, at least for the time being, any threat of mass resignations.

State Department spokesman Hodding Carter, who was rumored Sunday night to be resigning, announced at his daily press briefing that he would remain and added: "I know of no one else in the Department of State who has any intention of leaving in the immediate future."

There was general expectation that some of the people at the assistant secretary level who came into the department with Vance and who were associated most closely with his policy ideas will leave in the weeks and months ahead. But, in deference to appeals for a show of unity, their departure probably will be gradual and staggered.

At the White House, there appeared to be a sense of great sadness but little or no bitterness over Vance's resignation. Powell said Vance's action "was not related to whether or not it [the mission] would be successful," but resulted from the secretary's opposition, as a matter of principle, to any kind of military action, regardless of its chances of success.

The press secretary said that when Carter gave the initial go-ahead to the mission on April 11, the president was aware of Vance's feelings but did not know that it would lead to his resignation. Vance, who was away from Washington on a brief vacation on April 11, voiced his opposition to Carter in a private meeting after his return and again at a meeting of the National Security Council on April 15, but did not mention resignation on either occasion, Powell said.

Throughout this process, Powell said, "the president simply remained unpersuaded" by Vance's arguments; and it was sometime between the NSC meeting on April 15 and April 21 that Vance told Carter he would have to quit.

The president did try to persuade Vance to reconsider, and White House sources said there initially was a felling there that the matter could be resolved. But, as Powell said, reflection led to the conclusion that "it just wouldn't work" because of Vance's inability to support Carter on so fundamental a question of policy.

Asked about the political consequences, Powell replied, "A lot depends on the president's choice of a successor." He cautioned that the decision won't necessarily be made by tonight's news conference, but said Carter wants to avoid a prolonged Senate confirmation hearing and predicted a replacement will be named "in a matter of a few days."

Speculation about Christopher, deputy secretary under Vance for the past 3 1/2 years, centered on the fact that the two men share an identity of views on most issues. Christopher also is given high marks within the administration as a loyal team player and a man able to get along well with Congress.

His chief drawback is considered to be a cautious and colorless personality that many critics think would translate into a lack of forcefulness and allow Brzezinski to gain unchallenged status as Carter's principal adviser on foreign policy.

Still, there was a general feeling that Christopher would be the logical choice in terms of gaining quick Senate approval, keeping defections from the State Department to a minimum and reassuring the allies that someone with ideas akin to Vance's will still be present in the administration's highest councils.

At the meeting with his key aides yesterday, Vance said he personally hoped the nod would go to Christoper.

Of the other names being mentioned, most speculation tended to rule out Brzezinski on the grounds that he is regarded as too hawkish by allied governments, has a combative personality that would not mesh well with Congress and, by moving into the secretary's office at this time, would keep alive the controversy about a power struggle.

In response to questions yesterday, Brzezinski's office pointed to remarks he made publicly March 8, saying he had "never aspired to be secretary of state" and was "very happy with the job which I have."

Many sources said they felt Linowitz, who ordinarily would have strong backing, would be ruled out because he is involved in the delicate, make-or-break phase of the Egyptian-Israeli negotiations on Palestinian autonomy and could not be pulled off that assignment easily at this time. Cutler's chief drawback was described as the fact that he is a relative newcomer to the foreign policy arena and would have to undergo something of an on-the-job learning process.