As Secretary of State Cyrus R. Vance leaves today for Peking, U.S. officials rule out any quick breakthrough on the prickly political issue of switching American diplomatic recognition from Taiwan to the People's Republic of China.

Instead, in the Carter administration's first top-level encounter with China's new leaders, the priority is on developing a pattern of expanded global consulation with China.

If the conferees can build a basis for broader relations, officials here said yesterday, they can overcome the central obstacle between Washington and Peking. This is Peking's insistence that the United States end diplomatic recognition of the Nationalists on Taiwan as the government of China, nullify the 1954 Mutual Defense Treaty with Taiwan, and withdraw the remaining 1,165 U.S. troops from the island.

The American insistence is that the United States "will not abandon Taiwan." The area for compromise, which neither the Nixon nor Ford administrations could achieve, is substitute arrangements for the security and economy of Taiwan which Peking's Communist rulers would not challenge by military force or pressure.

This kind of solution, administration sources said yesterday, can only be achieved as "a process," extending beyond any one round of meetings, and requiring reciprocal actions by Peking, and its recognition of American political realities.

What this means in blunt terms is that the administration, currently engaged in an uphill battle to win support of two controversial Panama Canal treaties, does not want a simultaneous conflict with many of the same bitter conservative opponents of a break in U.S. relations with Taiwan.

At the same time, the Carter administration is anxious to improve its global and bilateral relations with China which rules 800 million to 900 million people, at some increased cost to ties with Taiwan and its million people, 100 miles off the coast of China.

The process of "normalizing relations" with China, begun by the Nixon administration in 1971-72, has been stalled for four years, partly because of political turmoil in both nations.

Administration officials say they are genuinely uncertain about what will come out of Vance's four days of talks which begin Monday.

China's new leadership which has just completed a Communist Party Congress is believed to be more eager to expand economic and other relations with the outside world. But the two pillars of Chinses authority with whom the United States ended a generation of hostility. Mao Tse-tung and Chou En Lai, are both dead. Some specialists suspect that Peking's new leaders may have less flexibility for compromise on the great symbolism of China's claim to Taiwan as one of its provinces. This is the single issue on which both Taiwan and Peking agree: that Taiwan is part of China.

As a consequence, the Carter administration has gone to unusual lenghts to label Vance's mission an "exploratory trip." Officials are determined to avoid any optimistic prediction that might backfire in headlines of setback, or failure, which followed Vance's trip to Moscow last March and his trip to Arab-Israel capitals that ended a week ago.

White House press secretary Jody Powell said yesterday, "This trip is to a major extent exploratory." he said, "We are not going there with major new proposals looking for, or expecting, some dramatic breakthrough."

As a degree of assurance to Taiwan, State Department spokesman Hodding Carter III said yesterda that, "We have remained committed to that nation's security and we have no plans to abandon it."

The Nationalist Chinese Ambassador to the United States, James C. H. Shen, had one of his infrequent meetings on Thursday with senior State Department officials. He met with Under Secretary Phillip C. Habib and the Assistant Secretary for East Asian and Pacific Affairs, Richard Holbrooke, who both will accompany Vance.

Vance had a lengthy discussion yesterday, the second in two days, with Soviet Ambassador Antoliy F. Dobrynin.

The Soviet Union has more than casual interest in Vance's visit to its bitter ideological rival. The Dobrynine discussions, however, were said to by part of a series to prepare for Vance's meeing in Vienna on Sept. 7 with Soviet Foreign Minister Andrei A. Gromyko on the obstacles to accord on a new U.S.-Soviet nuclear strategic arms limitation.

TheUnited States and China see each other as global counterweights to the SOviet Union, following parallel policies in many regions to checkmate Soviet objectives, although that view is officially disclaimed.

One major subject for discussion in Peking, officials said, will be southern Africa, where these interests converage. Other topics: the Middle East; Korea, Southeast Asia, the Indian Ocean area, and a review of U.S.-Soviet relations. Officials said the Vance mission will be explaining the entire pattern of Carter administration policy, including its human rights objectives. In the administration's view of the world, these officials said, there is full recognition of China as a global power with many matching objectives.

If China agrees to broaden this foundation ofshared interests, administration officials said, each side will show greater sensitivity to the other's problems - notably dealing with the several sources said, could then be in position to take on the formidable domestic task of seeking broader congressional support for transferring official U.S. diplomatic recognition from Taiwan to Peking, replacing it by "unofficial" links with Taiwan, as Japan, West Germany and other nations have done.

One senior American official suggested alternatively, establishing irplace of a U.S. embassy a Taiwan "liaison office" such as the United States and China set up in each other's capitals in 1973, Peking's acceptance of such official representation in Taiwan, however, appears improbable. One leading non-governmental specialist on China, A. Doak Barnett of the Brookings Institution, said the key is "what Peking will tolerate" to show the United States "is not bugging out on Taiwan."

Former Senate Republican leader Hugh Scott (Pa.) earlier this wek made public a 1976 China-trip report to iormer President Ford, saying "ther is very wide congressional and public support for rapid movement towards normalization of our diplomatic relations with the People's Republic of China, even at teh price of severing diplomatic ties wit our friends on Taiwan."

Sen. Edward M. Kennedy (D-Mass.) on Monday urged a similar course on President Carter, also with proposals for retaining reduced links with Taiwan. Kennedy's proposal was bitterly attacked by Sen. Barry Goldwater (R-Ariz.), who said it would "dishonor America" and "would nullify 59 treaties and agreements" with Taiwan, ranging from shoe import quotas to safeguards for nuclear materials.