The Vatican and the Kremlin have been fighting for men's minds ever since the world's first Communist state came into existence in 1917.

In Moscow, the Kremlin's massive walls topped with red stars act as the symbolic hub of monolithic Soviet-style communism. At the Vatican, Bernini's splendid columns topped by the saints of the Roman Catholic Church provide a spectacular public forum for Christ's "vicar on Earth," Pope John Paul II.

This week the contest between the material ideology of Marxism and the spiritual appeal of religion again assumes center stage as the first pope from a Communist country pays a return visit to his homeland. The lives of an estimated 70 million Catholics in the Soviet Bloc could well be affected by what Pope John Paul says and does in Poland during the next few days--and the way Moscow reacts.

What gives the pilgrimage a quality of suspense is the personal, almost mythical rapport that exists between the Polish pope and his countrymen. This makes the consequences extremely difficult to predict, just as it was impossible to foresee that one of the results of the pope's first visit to Poland in June 1979 would be the rise and fall of an independent trade union movement called Solidarity.

Until the late 1950s, the rivalry between the Kremlin and the Vatican was expressed in bitter ideological warfare. In the 1960s and 1970s, it was muted as each side recognized that it had more to gain by quiet negotiation than open confrontation.

Church officials and foreign observers here said they are convinced that the nature of the Vatican's relations with Eastern Europe--what has come to be called the Vatican's ostpolitik--has gone through a further twist since Pope John Paul's election in October 1978.

"Previous popes treated Eastern Europe as a diplomatic problem to be tackled by traditional diplomatic techniques. For Pope John Paul, it is an extremely emotional matter that goes right to the heart of his own private identity," a western diplomat accredited to the Vatican remarked.

In an interview with Washington Post correspondent Bradley Graham in Vienna, Austrian Cardinal Franz Koenig said the present pope has paid much more attention to questions involving religious freedom than Paul VI and has also sought more contacts with Eastern Orthodox churches.

"He knows very well that you can't change the Marxist system in the East. . . . The great thing is dialogue, and that is the key word for this pope. His predecessors started it and he broadened it," said the cardinal. As head of the Vatican's secretariat for nonbelievers, Koenig played a major role in the shaping of the Holy See's foreign policy in the 1970s.

The Vatican's ostpolitik dates to the death of Pius XII, a fervent anticommunist, in 1958 and the subsequent great opening up of the church under John XXIII. The major boost came with the convening of the Vatican Council and the first tentative contacts between the Vatican and individual Soviet Bloc governments in 1963.

At first, church diplomats saw the change as a means of achieving concrete goals. In 1963, for example, John XXIII arranged for the release from a Soviet prison of Cardinal Josyf Slipyi of the Ukraine. In return, he tacitly promised not to make an issue of the rights of the Eastern church.

"The original ostpolitik was an attempt to strengthen the formal structure of the church by removing some obstacles. Today there is much more feeling in the Vatican for considering the entire good of the people and not just for saving the hierarchy," said Michael Bordeaux of Britain's Keston College for the Study of Religion.

Pope John Paul's view of the Soviet Bloc has been shaped by his experience as a bishop and then as a cardinal in the southern Polish city of Krakow, one of the jewels of medieval Europe.

"What the pope would like to see is a return to the golden age of Charlemagne. He sees Europe as a single cultural, philosophical and spiritual entity, with Poland and particularly Krakow at its center. His Europe extends even to the Soviet Union which, by the way, he always calls by its historical name of Russia," a senior western diplomat at the Vatican remarked.

This vision of Europe, which the pope developed during his past visit to Poland, is at odds with the Kremlin's geopolitical interest in keeping the continent divided into two ideologically opposed camps. It helps explain the enormous suspicion with which Soviet leaders regard Pope John Paul.

Some observers make the case that the pope's last visit to Poland was too successful because it gave rise to expectations that could not be met. Change came too quickly and too suddenly for the moribund political system to absorb.

Hans-Jacob Stehle, a West German journalist who has studied the Vatican's relations with the East Bloc for 13 years and has written a book about it, said he believes that the pope originally wanted to show that it was possible to reform the communist system without destroying it. This concept was at the heart of Solidarity's strategy as well, but it suffered a severe setback with the imposition of martial law in Poland in December 1981.

"The pope will now be obliged to fall back on the conventional methods of Vatican diplomacy: negotiations, advance and retreat, a step-by-step approach," Stehle said.

The Holy See's foreign policy machine is housed on the third floor of the Vatican palace. It is called the Council for the Public Affairs of the Church and consists of only about 30 seasoned clerics. As an arm of the secretariat of state, it forms part of the pope's private household.

The business of state is conducted along corridors decorated with medieval maps of the world and delightful, if somewhat incongruous, frescoes by Raphael depicting cupids cooing at each other in the clouds. When he drew them in the early 16th century, the painter was wildly in love with a cardinal's niece.

This minute scale and informal setting make the council quite different from other foreign ministries, with their giant bureaucracies that tend to develop lives and institutional views of their own. About 80 percent of the reports and telegrams circulated in the council end up on the pope's desk. Information flows upward, rarely downward. For countries such as Poland where he has a special interest, the pope virtually acts as his own desk officer.

The result is that the character of the pope has a decisive influence on the shaping of policy. Paul VI was trained as a diplomat and had a bureaucrat's habits. Some Vatican officials still remember fondly how any memo submitted to him for approval in the evening would come back neatly annotated the next morning.

The present pope has little interest in paperwork. As a church official who has observed him closely said, "He sees his primary mission as pastoral, and it's as a pastor that he is going back to Poland."

The Polish influence at the Vatican has grown considerably since John Paul, formerly Cardinal Karol Wojtyla, became pope. Signs in St. Peter's Basilica have all been translated into Polish, in addition to English, French and Spanish. There is a special Polish- language edition of the Vatican newspaper, L'Obsservatore Romano. Poles occupy key positions in the Vatican hierarchy, and foreign powers now tend to select diplomats with expertise in Poland or in the Soviet Bloc to staff their missions to the Vatican.

The Holy See's relations with Eastern European countries cover a wide spectrum from correct to nonexistent. Vatican officials regard Czechoslovakia as the mouthpiece for the most virulently antichurch and antipope sentiments in the Soviet Bloc. Authorities in Prague have refused to agree to the Vatican's nominations for seven bishops and have harassed believers.

By contrast, authorities in Hungary are much more open in their approach to the church and quickly settle outstanding problems. The Soviet Union has allowed the Vatican to nominate a cardinal and several bishops in Latvia, although it follows a much more restrictive policy in other regions, notably the Ukraine.

"There's been a general cooling in relations since the events in Poland and as a result of the allegations about a Bulgarian connection in the attempt on the pope's life," a Vatican official said. "Communist regimes have adopted a wait-and-see approach toward the church, and in the case of Czechoslovakia, outright hostility."

He added, "You must remember, however, that this is not a totally monolithic bloc we are dealing with. Each country is very different and has its own seperate cultural, religious and economic characteristics."

One of the results of this week's pilgrimage could be a slight upgrading in relations between the Holy See and Poland. There has been some discussion in the Vatican about appointing a permanent nuncio to the church in Poland who would also have contacts with the Polish government.

Such an arrangement would be one step short of full diplomatic relations. It would reciprocate the present setup at the Polish Embassy in Rome where there is a special section headed by a former minister for religion, Jerzy Kuberski, in charge of contacts with the Vatican.

A Polish priest who works in the Vatican said the theme of this week's papal pilgrimage will be much different from that of the previous visit.

"The first visit led to a great national reawakening," the priest said. "The situation in Poland is completely different today. What people expect is an authoritative answer to the question of how to live honestly in a situation that seems hopeless and degrading. What are the limits of compromise with the present regime?"

It is acknowledged in the Vatican that the Polish authorities would like to use the visit to regain international respectability. The pope will be meeting Gen. Wojciech Jaruzelski and other Communist leaders, but he is likely to make clear that he regards his primary mission in Poland as providing moral support and encouragement for his fellow Poles rather than negotiating with the government.

As the Polish priest noted, "Whenever the pope arrives in a country, he first makes some gesture to the people such as kissing the ground before greeting the government leaders. You can be sure that he's going to make that distinction very obvious in Poland."