The staff of Le Monde, the newspaper of the French establishment and one of the central institutions of French society, elected a new editor today after months of politicking, balloting and inconclusive voting.

The winner was Claude Julien, 55, a Catholic leftist who has written a string of Anti-American and pro-Castro books.

Before the final vote, many Le Monde Staffers expressed fears that the election of Julien, who has been the front-runner since February, would push one of the world's most influential papers from a mild, intellectual form of anti-Americanism to strident advocacy of Third World causes.

The latter has been the dominant tone of the monthly Le Monde Diplomatique supplement Julien has been editing since 1973, losing his position as the paper's foreign editor after he mounted an unsuccessful challenge to the present moderate editor, Jacques Fauvet, and his deputy, Andre Fontaine.

"No matter what words of appeasement he may have uttered to win the job, Julien is a man with lots of old scores to settle," said a French journalist who claimed that three Le Monde staffers had told him separately they would quit if Julien were elected.

He is the first new editor to be chosen by Le Monde's 200-member editorial staff. Julien has had a simple majority for some time, but several efforts against him had prevented him from getting the 60 percent needed to win.

The merits of the rival candidates were discussed both inside and outside of Le Monde in terms ordinarily reserved for papal candidates. "The entire French press," said one paper after an earlier vote, "was waiting to see the rise of white smoke marking the end of a happy conclave, anxious to know if it were to have a new pope."

Julien and all of his serious rivals emerged from Le Monde's foreign staff, whose coverage sets the paper's general tone. After trying several times to overtake Julien's lead, the foreign editor, Jacques Amalric, withdrew from the race.

Julien's partiality to Third World positions could strengthen Le Monde's position as the only influential Western newspaper in much of Africa and the Middle East. It is also the most closely read newspaper of the elites trained in Paris universities. Approximately 20 percent of Le Monde's total circulation of 500,000 is abroad, mostly in the Third World.

Le Monde is the only financially healthy Paris daily and its staff is among the best paid.

Julien does not take over the editorship until Fauvet retires Dec. 31, 1982. He has pledged to respect the spectrum of opinions represented by Le Monde in its wide variety of contributors.