The United States has "a major opportunity" to set the stage for resuming full diplomatic relations with China in 1978 during the trip to Peking next week by Secretary of State Cyrus R. Vance, Sen. Edward M. Kennedy (D-Mass.) said yesterday.

Kennedy, in a Boston speech, outlined a formula for switching diplomatic recognition to the mainland from the Chinese Nationalist government on the island of Taiwan, while maintaining "unofficial relations" with Taiwan for its security and economic support.

The Kennedy is believed to contain many of the same ideas that Vance will explore in Peking. Kennedy associates denied that the senator was sending up "a trial balloon" for the Vance mission. But they said they had discussions with specialists inside and outside the government in drafting Kennedy's speech.

Kennedy told the World Affairs Council of Boston that, with new leaders in Washington and Peking, it is now opportune to complete the process begun in 1972 for establishing full diplomatic relations with the Communist rulers of China. The two nations established liasion offices in each other's capitals in 1973, as halfway houses toward opening embassies.

For its global interests, Kennedy said, the United States should accept Peking's demands for breaking official ties with Taiwan.

"We must end our diplomatic presence there," Kennedy said, "our defense treaty, and our formal diplomatic relations with the island."

This does not mean "abandonment of Taiwan," he contended. Neither, he said, is it "necessary nor useful to try to extract an explicit renunciation of force [against Taiwan] from Peking," as American diplomacy has sought to do as a condition for formal recognition of China.

Instead, Kennedy said, through "creative diplomacy" the United States can establish adequate assurance for the security and well-being of Taiwan's more than 16 million people.

The 1954 mutual defense treaty with Nationalist China, to assure the protection of Taiwan after the Nationalists were forced from the mainland in 1949, Kennedy said, "would no longer have effect when we establish relations with Peking as the government of China."

Carter administration officials are known to have reached the same conclusion. A State Department legal opinion says the withdrawal of diplomatic recognition from the Taiwan government would cancel out that treaty, without need to give a year's notice of termination as specified in the treaty.

The United States, Kennedy said, "should make clear unilaterally our continuing opposites to use of force against Taiwan." In addition, he said, "unless and until" there is a peaceful settlement between Taiwan and the mainland People's Republic of China, "We should continue to ensure that Taiwan has access to supplies needed for self-defense."

Also, Kennedy said, "we should encourage the Chinese to indicate unilaterally that they will be patient and will continue to prefer peaceful means of reunification with Taiwan."

"The Chinese, in turn," Kennedy said, "should be expected to be sensitive to our interests and concerns."

This means, said Kennedy, that China would "not . . . oppose reasonable steps by the United States to provide for a prosperous and peaceful Taiwan."

When Vance goes to China next week, Kennedy said, the agenda for talks should include "an early date for complete withdrawal of our few remaining military forces . . . from Taiwan."

These personnel totaled about 10,000 when the Nixon administration began the process of normalizing relations with China, and are now down to 1,165, a Defense Department spokesman said yesterday.

The process of restoring full American-Chinese relations, Kennedy said, should start "as soon as possible, preferably no later than 1978." Before then, Kennedy said, the two nations should settle the unresolved question of financial claims against each other, increase official and unofficial exchanges, and expand trade.

To delay the improvement of relations with Peking, Kennedy said, can jeopardize the triangular relationship in which the United States gains advantage from the bitter rivalry between the Soviet Union and China.

China's "unremitting opposition to Soviet power" cannot be taken for granted indefinitely, Kennedy maintained, for "Chinese frustration with the United States over Taiwan might eventually overcome" the late Mao Tse-tung's "strong anti-Soviet legacy and lead over time to a limited detente between China and the U.S.S.R."

There was no immediate comment from administration officials on the Kennedy speech, which was jointly sponsored by the Boston World Affairs Council, the Fletcher School of Law and Diplomacy of Tufts University, and Harvard University's East Asian Research Center.

President Carter, who will be making his final decisions this week on the mission for which Vance departs on Saturday, said in a recent interview: "There is a chance we might have recognized the People's Republic" by this time next year.

The Vance trip, planned for months, will be the Carter administration's first high-level meeting with China's leaders. The administration is pledged to try to move toward normalized relations with Peking, a process which has been stalled since 1974 because of the problem of Taiwan, and by domestic events in both China and the United States.

"If I could write the script," President Carter said in a July 29 interview," I could devise a way out of the dilemma" over Taiwan. The U.S. goal, he said, is to normalize relations with Peking."But we don't want to be in a position of abandoning the commitment to the peaceful existence of the people of Taiwan."

There was no immediate administration comment on the Kennedy approach. The type of compromise he outlined is certain to face strong opposition in Congress and across the country.

A recent study by Potomac Association of Washington reports that 62 per cent of Americans think it is either "very important" or "fairly important" to establish full diplomatic relations with mainland China. "This majority evaporates, however, when Peking demands as a precondition that the United States sever diplomatic and military ties with the Republic of China on Taiwan," the study said.