A formal handshake between President Carter and Italian Communist Party leader Enrico Berlinguer at a state dinner given by Italy's president tonight was a measure of how far the party has come toward achieving respectability here, and also a signal of the Carter administration's increasing openness to dealings with the Italian left.

Berlinguer's encounter with Carter was his first with a top U.S. official. The Communist leader had not even met before with U.S. Ambassador Richard Gardner, although the envoy has talked informally with several other top party leaders.

The handshake also demonstrated that the party is willing to accept the risk of being tarred with the American brush in its continued drive for acceptance as a normal democratic party that should be allowed to participate in the Italian government.

The Kremlin put out warning signals that it considers the price the Italian Communists appear willing to pay far too high. In what was seen as a warning to Berlinguer not to go any further on his current pro-Western path, the authoritative Soviet foreign policy weekly New Times attacked his foreign affairs spokesman Giancarlo Pajetta.

"The world," said New Times, "is the arena off a formidable struggle between progress and reaction, between the forces of socialism and the forces of imperialism. On what side is Comrade Pajetta in this fight?"

Apparently in deference to U.S. sensitivities, no photographers were allowed at the dinner for the top 200 personalities on Italy's political scene.

The Carter administration accepted the reasoning of Italian President Sandro Pertini that it would be a breach of protocol not to invite the leader of Italy's second most important political party to the state dinner.

The last state dinner here for a U.S. president was when President Nixon visited Rome in 1969.

Then, at the height of the Vietnam war, the Italian Communists staged major demonstrations against the visit.

Italy's Communists have recently condemed the Soviet invasion of Afghanistan, refused to attend a major Soviet-engineered conference of European Communist parties in Paris and staged a Berlinguer-led visit to Peking -- Moscow's arch-rival in the communist world. After Yugoslav Marshal Tito's death, Berlinguer publicly warned the Soviet's against any attempt to alter Yugoslavia's "courageous policy of nonalignement" that is vital to Italy's security and the European balance of power.

The Italian Communists waged what seemed like only a symbolic parliamentary battle against the most important U.S. demand on the Italian government in recent months -- acceptance of the deployment in Italy of U.S. medium-range missiles that can reach the Soviet Union.

A top-ranking Italian diplomatic counselor said that his government had consulted the Communist leaders beforehand about the decision. Since, the Communists have openly aligned their position with the pro-U.S. West German policy.

Scattered Communist Party posters have appeared around Rome calling on President Carter to respect Italy's "autonomy." Italian party sources confirmed that the party's approach to Carter's visit has been deliberately low key. "We speak our mind, but we are gentlemen," said a high-ranking Communist.

The Italian Communist Party is a much more dynamic force on the world foreign policy scene than the Italian government itself. In addition to its dramatic public stands, the party is known to act frequently to influence the foreign policies of Third World countries away from the Soviets.

As a veteran West European ambassador here noted, "The Italian Foreign Ministry worries about two things when it takes a position -- what Washington thinks and what the Communist Party thinks."

Berlinguer's increasingly pro-Western line has not been without risks. In March, about 50 or 60 of the party's 190 deputies in Parliament voted against a resolution approved by their leaders favorable to the Italian government's pro-Western foreign policy. Although it was a secret ballot, there could be no doubt that there had been the first break in party discipline in Parliament that anyone could recall.

The Berlinguer line was vindicated earlier this month when the party actually did better than expected in nationwide regional and local elections, polling 31.5 percent of the voice and making some advances in the industrial north that compensated for losses in the depressed rural south. Berlinguer's pro-Soviet opponents inside the party had laid the groundwork for a challenge over foreign policy in case the party did poorly.

Nevertheless, in a White House background briefing Monday that got wide publicity here, Carter administration officials greeted the election results as part of a pattern of "important electoral setbacks" for the Communists, whose high point was 33.4 percent in 1975.

This commentary prompted an editorial yesterday in the Communist newspaper L'Unita denouncing Washington's "imperial and clientalist view of international relations."

Whatever pique there may have been, it was not great enough on either side to prevent a handshake that was worth a thousand words, even if it did not rate a single picture.