Former Secretary of State Henry A. Kissinger last night tried to shake the Carter administration out of its hands-off policy toward Communists winning power in Western Europe.

"The effect on alliance cohesion generally would be disastrous" if Communist parties gain "significant participation" in the governments of France, Italy other alliance nations, Kissinger warned in a major address here.

"In this process," Kissinger said, "it is vital that the United States encourage an attitude of resolve and conviction." He cautioned that "ostentatious association or consultation with Communist leaders" or "ambiguous declarations" of U.S. attitudes can create "the impression that we consider Communist sucess a foregone conclusion . . ."

The Carter administration has shifted away from the alarmists attitude toward Western Europe communism that Kissinger pursued in office. The U.S. position how amounts to a policy of non-interference.

Kissinger sough to reopen the debate, in a dinner speech concluding a Conference on Italy and Eurocommunism, at the Smithsonian Institution's Woodrow Wilson International Center for Scholars. Copies of the speech were circulated in advance.

As has been his custom since he left office, Kissinger avoided any direct attack on the Carter administration, and at one point he referred favorably to the President.

His theme, however, was: "We must not delude ourselves about what the accession of Communist leaders to executive power will mean to the most basic premises of American foreign policy. We must not confuse either our own people or those in allied countries who take our judgments seriously, about the gravity of the threat."

Kissinger also pointedly addressed one of the arguments turned against his former policy inside the government: that Americans sounding a Communist alarm to Western Europeans "would encourage Communist protest votes."

"I believe the opposite to be true," Kissinger said. "On balance, I consider it important that Europe know of America's interest and concern."

Present U.S. policy, as declared by the State Department on April 6, is:

"We believe that the position of a Communist Party in a particular country is a matter to be decided by the people and government of the country concerned. We do not propose to involve ourselves in the processes by which they reach their decision on it."

This does not represent an attitude "of indifference," the statement added, about dominance of allied governments by political parties with alien traditions and values.

President Carter said in a May 2 interview that "it's not up to us to tell other people how to vote," and that the United States strongly favors leaders "free from Communist philosophy . . ." But he said "the best way to prevent a shift toward communism in Italy or France or other countries is to make sure that the democratic government that's presently in existence works," open to change and sensitive to needs.

Kissinger last night said the United States has "a duty" to make clear the consequences "if Communist minorities gain decisive influence in European politics."

Much has been made, he said, of "gesture(s) of independence" by Western European Communist leaders, such as their renunciation in 1976 of "the Soviet concept of dictatorship of the proletariat . . ." But "now," he said, "it turns out that the new Soviet constitution, in preparation for years, drops the phrase as well."

The argument is raised, Kissinger said, that if the United States can deal with Communist governments in the Soviet Union, China, Cuba, Vietnam or elsewhere, why not with Communist parties in Western Europe?

Kissinger said, "There is a crucial difference between managing conflict with adversaries and maintaining an alliance among friends . . ." The key issue, Kissinger said, "is not how 'independent' the European Communists would be, but how Communist" they would be.