President Carter, moving swiftly to contain the foreign policy damage caused by Cyrus R. Vance's resignation, reached out to the Washington establishment yesterday and named Democratic Sen. Edmund S. Muskie of Maine as his new secretary of state.

Carter's choice was announced at a late afternoon ceremony in the White House press room, where Muskie pointly said, "The president has left no doubt in my mind . . . that I will be the foreign policy spokesman" for the administration.

The remark seemed especially significant because of the circumstances leading to Muskie's appointment -- Vance's resignation in protest against last week's abortive attempt to rescue the American hostages in Iran, and subsequent speculation that Vance had lost out to Carter's national security affairs adviser, Zbigniew Brzezinski, in the struggle for control of foreign policy.

In fact, the nationally televised White House ceremony seemed to have been staged almost as a tableau to reassure the American public that the administration's top polcymakers are closing ranks behind the president.

Standing behind Carter and Muskie on the podium were Vance and Brzezinski, both smiling broadly, as well as Vice President Mondale, Defense Secretary Harold Brown, whose Pentagon planners conceived the ill-fated rescue attempt, and Acting Secretary of State Warren M. Christopher, who had been widely expected to be designated as Vance's successor.

Both Muskie and Carter praised Vance lavishly for his accomplishments as secretary and expressed understanding for Vance's position that, because he disagreed with the concept of using force in the Iran crisis, he had to resign as a "matter of principle."

In 1972, when Muskie made an unsuccessful bid to win the Democratic presidential nomination, Vance, then a prominent Wall Street lawyer, was his chief foreign policy adviser.

Carter also made clear that Christopher had agreed to stay on as Muskie's deputy and referred to them as a team "that will provide continuity for our foreign policy." White House sources said the president personally had asked Christoper, who was Vance's choice to be his successor, to remain in the deputy post.

Despite this ritualistic display of unity and good feeling, Carter's choice of Muskie -- a man whose name hadn't been mentioned in all the speculation about Vance's replacement -- left unclear how much influence and authority the new secretary will have in the administration's pecking order.

According to several sources, the chief problem was alluded to by Brzezinski, in joking fashion, at a meeting in the White House shortly before the public announcement. Referring to the fact that he and Muskie both are Polish-Americans, Brzezinski told Carter: "Mr. President, I just want to assure you that Mr. Muskie and I will never be poles apart."

As his pun suggested, Washington as well as other governments will now be watching closely to see what kind of role Muskie carves out for himself and whether he will be overshadowed by Brzezinski in the battle for influence with Carter.

At a separate news conference last night, Carter said reports about Brzezinski's power were "erroneous" and insisted that he has a "very good and proper balance of advisers." Although the president said he would not allow anyone to make "an improper intervention" in matters that are the concern of the State Department, he stressed that he reserves the right to make the final decisions on policy.

With 22 years of Senate service behind him, Muskie is an important and influential figure on the Washington scene. He is no novice in foreign policy, having served a total of six years on the Senate Foreign Relations Committee, and he gained considerable additional knowledge of the subject when he ran unsuccessfully as the Democratic vice presidential candidate in 1968 and sought his party's presidential nomination in 1972.

However, his principal reputation, and his major interest in recent years, has been built on domestic policy issues, mainly those related to the environment and government spending. In fact, his departure from Congress will leave a big void in the leadership in the fighting over those questions.

There was an initial tendency last night to assume that, by turning to Muskie, Carter had pulled off a brilliant political stroke. According to this view, he came up with a man who enjoys national stature and has great influence with Congress, but who will have to go through a trial-and-error period before he is in a position to intervene authoritatively in the policy directions now being pursued by the president under Brzezinski's general guidance.

Yet Muskie is known both as a tough fighter and a man who knows how to wheel and deal in Washington power circles, and he could use the free-and-easy political style of his Senate years in ways that will provide surprises.

A hint of that was evident in his behavior at the end of yesterday's announcement ceremony. Although Carter and the other dignitaries moved off the podium following the formal statements, Muskie unexpectedly remained, batting comments back and forth with reporters to the obvious discomfort of White House officials in the background.

Explaining the choice of Muskie White House press secretary Jody Powell said, "There was a feeling we needed someone of stature who would project a general sense of continuity." Powell noted that Vance had advised Muskie on foreign policy issues when the senator was bidding for national office in 1968 and 1972.

Powell also said the administration sought a nominee who would quickly gain Senate confirmation, and someone "with whom the president felt comfortable."

Carter, at his later news conference, said he considered Muskie "extremely well qualified" because of his extensive experience, his knowledge of "the entire nation" and its aspirations, and his service as chairman of the Senate Budget Committee. In that post, the president noted, Muskie had been in the unique position of having to study and pass on the entire range of budgetary questions involving foreign and defense policy.

In his debut yesterday, Muskie did nothing that went against the grain. In answering reporters' questions, he was careful to confine himself to generalities about the "challenge" of the job and, on the Iran crisis, stressed his determintion to seek a solution through "peaceful means."

But, in a statement issued earlier in the day, Muskie referred to the controversial rescue mission in terms that appears to put him squarely behind Carter's approach to the Iran situation. He said:

"I believe the president had the authority and the responsibility to explore the feasibility of a rescue mission, to develop it as an option and to carry it out at such time as his best judgment dictated.

"Secretary Vance apparently disagreed. I respect his view, as do all Americans. But I believe most Americans also understand and support the president's decision . . .After all, it was a response to kidnaping and blackmail. We cannot rule out the use of any appropriate means which might end the confrontation and bring sanity back to Iran."

Muskie said he was in Nashville and unaware of Vance's resignation when Carter called Sunday night to offer him the post, and he promised to think it over. Powell said afterward that Muskie went to the White House Monday evening to inform the presient of his acceptance.

Carter provided Muskie with a government plane yesterday for a quick, unannounced trip to Augusta where he discussed with Democratic Gov. Joseph E. Brennan the question of his successor. Muskie said it would be up to Brennan to choose someone to fill out his Senate term, which expires in 1982.

Although Powell spoke of Carter's desire for "quick confirmation," Muskie faces one possible delaying problem. A constitutional provision says he cannot be formally nominated until the secretary of state's salary is reduced to the same level that it was when he began his current Senate term. So he must for Congress to pass legislation cutting the salary from $69,360 to $60,000 -- thereby rescinding a recent raise.

Anticipating the congressional reaction to the appointment, a Senate aide said, "The music will be out of 'Aida.' He's a very good, popular choice. No one up here is going to say Muskie isn't willing to stand up to a certain fellow countryman downtown [Brzezinski]."

This prediction was seconded by congressional leaders who were briefed by Carter at the White House before yesterday's announcement.

Typical was the comment of Senate Foreign Relations Committee Chairman Frank Church (D-Idaho), who called Muskie "a pretty tough guy" and said, "His nature leads him to be firm in his job. There is no doubt about his capacities.

Senate Majority Leader Robert C. Byrd (D-W.Va.) said he greeted the announcement "with genuine mixed feelings . . . because it comes as a loss for the U.S. Senate." But, Byrd added, Muskie is "the right man for the job at the right time" and said he "will place his individual stamp on foreign policy."

Sen. Jacob K. Javits (R-N.Y.), ranking minority member of the Foreign Relations Committee, also called the choice "very much an appointment for this season." Because of Muskie's experience and presitige on Capitol Hill, Javits said, "he comes to this job with an awful lot of clout."

Rep. Clement J. Zablocki (D-Wis.), chairman of the House Foreign Affairs Committee predicted that Muskie will ensure greater consultation with Congress on foreign policy. Sen. Richard J. Lugar (R-Ind.), citing Muskie's well-known temper, remarked: "There is an important time for anger. Ed Muskie is very good at using that."

In another development yesterday, after a two-hour closed meeting of the Senate Foreign Relations Committee, Chairman Church said that administration officials will be asked to appear next week to try to work out ground rules on future consultations with Congress.

He said no consensus was reached on whether lack of consultation before the Iran mission violated the War Powers Act, and said this question would be looked at in several weeks.