The State Department acknowledged yesterday that its officials had advance discussions with Sen. Edward M. Kennedy (D-Mass.) on his speech urging a shift of American diplomatic recognition from Taiwan to China in 1978.

Kennedy "was speaking for himself" in the formula he proposed on Monday for resolving the controversial issue, said spokesman Hodding Carter III.

"We were aware that the speech was coming," said Carter, and "the senator was among a number with whom we have had consultation of our China policy." Carter said he has no reason to believe that Secretary of State Cyrus R. Vance had "knowledge of the wording that the senator was going to use in his speech."

Kennedy's speech was neither "a joint venture nor a trial baloon," Carter said, for Vance's sensitive mission to Peking next week, to discuss the "full normalzation of relations" with the People's Republic of China.

Carter stressed that Vance's trip, which begins Saturday, will be "basically exploratory," without any expectations of "great results or pronouncements" from the Carter administration's frist top-level meeting with China's leaders.

In private, nevertheless, administration sources were pleased with Kennedy's speech because it opened up for public discussion what the administration itself does not want to say aloud. By neither confirming nor denying that Kennedy's speech pointed in the direction that U.S. strategy will go, administration strategists gain the advantage of testing reactions to a shift in China policy that will arouse great dispute.

The formula Kennedy urged for circumventing the thorny China issue was discussed at length, informed sources said, with specialists of the National Security Council staff and the State Department, over recent weeks.

The core of the approach set out by Kennedy would substitute "unofficial relations" between the United States and the Nationalist Chinese government on the island of Taiwan for diplomatic recognition, which would be shifted to the Communist-ruled People's Republic of China.

This would help assure the security and economic survival of Taiwan, Kennedy maintained, while meeting Peking's demands that on Taiwan "we must end our military presence there, our defense treaty, and our formal diplomatic relations with the island."

State Department spokesman Carter, while avoiding any of those specifics, reiterated what Vance said on June 29: "As we prepare to go to Peking, we recognize that progress" toward normalizing relations with the People's Republic "may not be easy or immediately evident."

The United States is committed "to move full normalization of relations" with Peking, in place of the present diplomatic liaison offices in both capitals, Carter said, but Vance can only explore "exactly where and how and when" the process may be accomplished.