Cyrus R. Vance, man of the establishment, has decided to quit the establishment on a point of priniciple -- an idea that has grown unfashionable in the corridors of American power that Vance walked with the confidence of a most valuable player.

Those closest to the secretary of state have known for months of his deep pessimism about the course of the Carter administration's foreign policy, and particularly pessimism about the influence on the president of Zbigniew Brzezinksi, Carter's national security affairs adviser.

But even closer friends were surprised that Vance had gone so far as to quit the job for which he spent most of his life preparing. "Isn't it magnificent?" one friend said last night.

The same friend had speculated sadly, just a few days ago, that Vance was likely to go into the history books as a mediocre secretary of state precisely because he would never be able to force a showdown between himself and Brzezinski with Carter, or to openly opt out of the foreign policy that he knew he could no longer control .

Vance confounded that analysis -- one widely shared by people who knew him well -- by deciding last week to resign his job. It was probably the first unpredictable thing Vance has done in his 63 years.

In the Johnson administration, which Vance served as a loyal field officer, he was typical of a large number of senior officials who privately held reservations about the war in Vietnam, but never jumped ship. Vance at least had a chance to work for peace in Vietnam as W. Averell Harriman's deputy at the initial round of Vietnam peace talks in Paris.

When Richard Nixon brought the Republicans to power in Washington, Vance retreated whence he had come, to the Wall Street Law firm of Simpson, Thatcher & Bartlett.

But this was a tactical retreat. Vance threw himself into the life of a secretary-of-state-in-waiting. He was active at the Council on Foreign Relations, at the Ditchley Foundation's international meetings in England, in Democratic Party policitics.In April 1976, when many of his potential rivals were still waiting coyly to pick a candidate in the jumbled Democratic race that year, Vance picked Jimmy Carter.

At first he had to explain himself to friends. But soon afterward the Pennsylvania primary confirmed Carter's status as the likely nominee, and Vance suddenly looked prescient.

Vance was never a domineering figure. He won the affection and admiration of the foreign policy establishment by force of decency and diligence -- he was admired for the quality of his preparation for the job, not for his intellectual creativity or the power of his personality. When Carter chose him to be his secretary of state, the sighs of relief were audible, both in America and overseas, particularly in Europe and the soviet Union.

In office Vance assembled a team of younger people, all with reputations for high-quality performance in the trenches of Washington's foreign policy guerrilla war. He salted that group with outsiders, many of them veterans of the carter campaign, including his personal spokesman, Hodding Carter.

Vance was determined to show that one could be secretary of state without being Henry Kissinger. His notion was not to globetrot, but to reestablish the State Department and its traditional diplomatic methods as the focal point of American foreign policy.

But this plan was foiled, in part by world events, in part by a discomforting aspect of the new Job for which Vance was not prepared: competition with Brzezinski.

The national security affairs adviser -- who did regard Kissinger as a model, not for the secretary of state but for his own new position -- had a strong view of world affairs, and it was not Vance's view. By instinct Brzezinski was competitive, confrontational. Though he had written in previous years about the need to diffuse the tensions and preoccupations of the Soviet-American competition, it was precisely that arena that Brezezinski in power found most congenial.

On paper Vance had an enviable position. Every evening he wrote a personal letter to President Carter summarizing has day's activities, and putting forth whatever views he though the president should have. This letter went directly to Carter, with no intermediary stops.

But Brzezinski had an even better position -- he began the president's day, every day, with a briefing on world events. By all accounts, Brzezinski seized this opportunity to paint and repaint a grand global view for the unschooled new president, a man of Georgia whose experience in foreign affairs consisted of his participation in a few meetings of the Trilateral Commission (for which Brzezinski had been chief factotum).

Vance, aides said , dealt with Carter in terms of specific issues, following his natural instincts as an attorney who sought to isolate and resolve specific disputes. His approach was no contest for Brzezinski's grandiloquent descriptions and analyses, or so many of Vance's friends and colleagues came to believe.

But for the first year or more of the Carter administration. Vance prevailed on any number of issues. He rarely if ever lost a direct confrontation with Brzezinski on a specific policy matter. He helped shape the early administration policy on the Middle East, on the Third World and as specific international disputers.

White House aides who watched this Brzezinski-Vance contest from a position of sympathy for Vance said then -- in 1977 and early 1978 -- that Carter sided with Vance because he thought like Vance, that the president's natural inclinations paralleled Vance's. The theme then was to avoid confrontations, to look for solutions and not for fights.

A turning point came in the spring of 1978, after the Soviets' successful use of Cuban surrogates in Ethiopia, when a bizzaree rebellion broke out in Shaba province of Zaire, the former Belgian Congo.Brezezinski and others interpreted this as a grave turn in events for the United States, and sought a firm American response. Vance demurred.

Soon afterward, Brzezinski went to China in the Carter administration's first effort to solidify the Nixon administration's opening to China. The atmosphere created by events in Africa fueled Brzezinski's combativeness; he was publicly quoted in China as making provocative anti-soviet remarks. When he returned to America, Brzezinski went on "Meet the Press." In that forum he made the strongest statements yet heard from the Carter administration on Soviet behavior, warning that Soviet activities in Africa were not "compatible with what was once called the code of detente."

For Vance this was a dismaying spectacle to Carter. The precise exchange between them has never been revealed, but according to the best accounts, Vance told Carter that he would have to choose between Brzezinski's hard line and his own cooler approach. In June 1978, at one of the regular breakfast meetings of senior foreign policy advisers, Carter opted for Vance. He told his aides he wanted the secretary of state to be his principal spokesman on foreign affairs. For many months, Brzezinski's star was in eclipse, and Vance's rose.

It was last summer when Vance's position again began to slip. A key event, no doubt, was Vance's offhand remark to reporters on an airplane flying around the Middle East that he had no intention of serving a second term as secretary of state. This was not inconsistent with his earlier views on his probable tenure, but the press made it a big story, and Vance suddenly took on the appearance -- and the weaknesses -- of a lame-duck secretary.

Subsequent events seemed to play into Brzezinski's hands. The "discovery" of Soviet combat forces in Cuba, then the seizure of American hostages in Iran and the Soviet invasion of Afghanistan recreated a confrontational atmosphere in world affairs.

Political pressures mounted on Carter to "do something," to show toughness and resolve. Brzezinski was full of ideas for such endeavors; Vance counseled persistently against undue adventures. Carter, by then a weak president facing a serious challenge from Edward M. Kennedy, was drawn increasingly toward Brzezinski's views. Vance and his colleagues felt themselves being pushed back into the cold.