At the massive gathering here this week of international literary talent -- a group that looks as much like the ultimate Famous Writers School as it does the 48th annual PEN Congress -- it was inevitable that politics, as much as writing, would be on the agenda.

And it was inevitable, too, perhaps, that Norman Mailer, even in his rather tweedy role as president of the American Center of PEN and host of all this action in the hotels along Central Park South, would be at the center of the maelstrom.

Ah, Stormin' Norman! Calmness does not follow him. Without consulting the PEN board of directors, Mailer invited Secretary of State George Shultz to speak at tonight's opening ceremonies at the New York Public Library. Mailer said the ceremonies would be "dignified" by Shultz's presence, but a number of other writers, PEN board members among them, did not agree.

Earlier today more than 60 of them signed and circulated an open letter to Shultz, criticizing the Reagan administration for doing "nothing to further freedom of expression, either at home or abroad," but Shultz spoke tonight with only minor heckling, saying, "Don't be surprised by the fact that Ronald Reagan and I are on your side."

As to the controversy surrounding his presence, Shultz said, "In Norman Mailer's world, it's a high form of flattery. And that's how I take it. I salute you for this defense of free speech."

After the speech, Grace Paley stood and demanded that Mailer read the open letter, but he refused. There also had been some protest outside the library tonight -- some over Shultz's appearance, but most over the fact that it took almost an hour to get everyone past a security check and into the building.

The controversy began last week, when E.L. Doctorow made his anger over Shultz's selection public in an article for The Nation.

"It is difficult for me to understand why he was invited," Doctorow wrote. "What has Shultz written? What is his connection to the world of letters? . . . Has he ever as Secretary of State championed the cause of universal free expression that so concerns the international community of writers?"

Kurt Vonnegut, who, like Doctorow, is on the board of directors, said in an interview that while Mailer's invitation was "a parliamentary mistake, the only real option we had was to disinvite Shultz. We voted not to do that. I don't think it's really as important as Doctorow does. I felt, what the hell, it can't matter that much."

Gay Talese agreed with Mailer and Vonnegut. "PEN should set an example for the freedom of expression," he said. "To deny our forum to anyone is appalling." Talese said he expects further debate when UNESCO Director General Amadou Mahtar M'Bow, arrives at the congress Tuesday. "But I can't believe anyone would want a closed-door policy," Talese said.

So as it turned out, Shultz mingled at the library with nearly 700 writers from 45 countries, including Jorge Amado from Brazil, Kobo Abe from Japan, Wole Soyinka from Nigeria, Omar Cabezas from Nicaragua, Gu nter Grass from West Germany, Eugene Ionesco from France, George Konrad from Hungary, Wang Meng and Lu Wenfu from China, Danilo Kis and Vasko Popa from Yugoslavia, Ryszard Kapuscinski and Wislawa Szymborska from Poland, Tom Stoppard and Salman Rushdie from England, Nadine Gordimer, Breyten Breytenbach, Sipho Sepamla and J.M. Coetzee from South Africa.

For the last decade PEN has actively protested against, among other things, the "ideological exclusion" provision of the McCarran-Walter Act, which may be invoked to prevent writers with present or past radical affiliations from entering the country. Although the State Department waived the "excludability" status of several writers, including Abe, Kis and Cabezas, at least one writer, Graham Greene, refused to come at all because of the McCarran-Walter law. Nobel Prize-winning novelist Gabriel Garcia Marquez, who has had difficulties with the McCarran-Walter provision in the past, is notably absent.

"I'm not surprised," said Vonnegut. "These people are insulted by that law."

Shultz said in his speech that the State Department does its best not to exclude any people because of their "abstract beliefs," but will continue to bar those whose actions "undermine" the U.S. government. "These judgments are made in good faith," he said.

PEN was established in 1921 in England by Nobel Prize winner John Galsworthy and Amy Dawson Scott. The next year Anatole France assumed the presidency of a French PEN Center, and Booth Tarkington did the same in the United States. Now there are 82 PEN centers in 62 countries. The American Center, with 2,000 members, is the largest, and is active in campaigning for the freedom of writers censored or imprisoned in a number of countries, especially the Soviet Union and Eastern Bloc nations.

Beginning in 1934 with H.G. Wells, American presidents of PEN have tried to establish a chapter in the Soviet Union. Instead, that country has been aggressively anti-PEN, shutting down the center in Latvia when Moscow invaded in 1940. Likewise, the Polish government shut down the PEN Center in Warsaw after the uprising of the Solidarity movement.

Mailer and PEN Center Executive Director Karen Kennerly invited eight Soviet representatives, including novelists Danil Granin, Valentin Rasputin and Chingiz Aitmatov, to the congress. Mailer and Kennerly wrote to Genrikh Borovik of the Soviet Union of Writers three times and even visited Soviet Ambassador Anatoliy Dobrynin, but so far, nothing.

"The last we heard, the Soviets were still thinking about it," said Kennerly. "We'd hoped that the U.S.-Soviet cultural accords signed at the Reagan-Gorbachev summit would have helped, but we'll have to see. It looks pretty bad now." Vassily Aksyonov, author of "The Burn," is a Soviet e'migre' now living in Washington. He will participate here this week.

Through private and corporate donations and a series of eight public readings given by Mailer, Alice Walker, John Updike, Woody Allen and others, PEN raised $800,000 for the congress. Its theme will be "The Writer's Imagination and the Imagination of the State," and the week will be dominated by a series of seminars on such literary-political topics as "How Does the State Imagine?," "Alienation and the State" and, on Friday, "The Statesman's View of the Imagination of the State," featuring Roy Jenkins, Pierre Trudeau, Bruno Kreisky, Carlos Fuentes and Vonnegut.

After apologizing for the occasionally aggressive reception accorded Shultz, Mailer was at his best tonight, speaking on the theme for this year's congress:

"When states begin to perceive themselves as protagonists, that is to say as embodiments of the creative vision, we may be entitled to speak not only of the imagination of the state, but to perceive of such states as actors in a scenario, or characters in a novel . . .

"Some kind of odd, even deadening, imagination seems to be at work, some unseen species of cooperation to flatten our spirit, some flatulent compact in world esthetics, some unspoken cooperation between nations otherwise at raw odds, to leech out the culture of the world. It is cause to wonder if the nuclear holocaust that may yet destroy us can come into being only after we have deadened the ecology, the esthetics, the culture and the meditations of silence sufficiently for life to become so harried and hectic, so consumed by unrest and static, that existence will have become less desirable than a final conflagration."

Before all such gatherings there is always some speculation as to what exactly can, or should, be accomplished. In a recent article for The New York Times Book Review, Susan Sontag, an active PEN member, said such meetings are "generally dismissed as tiring, often tiresome, an amiable waste of time. At best, a pious duty."

Even the most ardent PEN congregants agree these meetings are not all wit and imagination:

"I was at the PEN Congress in London a decade ago," said novelist Elizabeth Hardwick, "and I must tell you I didn't value it at all. It was an awful lot of people from small countries giving bad speeches on truth and beauty."

Vonnegut said, "It's true, these things are quite routinely a bore. Writers are, by definition I guess, not speakers. But it's important the congresses take place. They're valuable for the friendships they form. I met Heinrich Boll in Sweden, Alain Robbe-Grillet in Tokyo last year.

"Also, the gossip is extremely interesting. You learn about your fellow writers. I mean serious gossip; the way, for example, it's hell to be a writer in Poland. So a lot of it is boring, but not all of it."

Talese, who is at work on a sort of Italian "Roots," said, "The potential of a congress like this is tremendous. All of these writers, who produce their work in solitude, can come together and exchange ideas, exchange their presences in one place. Sure it can be boring or long-winded at times, but the chance for edification is tremendous. The experience is enlarging. It's just too bad that this all had to start on such a negative, fractured note."

Frances Fitzgerald, author of "Fire in the Lake," said, "The American reading public is pretty provincial. They concentrate mainly on Americans and some British authors. I'm hoping this sort of meeting will focus a lot of attention on writers from other places."

Although the theme of the congress will be highly political, Mailer said he wants to avoid "invidious comparisons" between countries.

"PEN was founded on the attractive notion that writers speak across national boundaries more gracefully and instinctively than governments," he said. "So when they get together there is, one may hope, a real possibility that new solutions, even surprisingly creative solutions, can be found.