Passing over best-selling and highly praised books by Norman Mailer, William Styron and Philip Roth, the National Book Critics Circle last night chose Thomas Flanagan's "The Year of the French" as the best novel of the year.

The nonfiction prize went to Telford Taylor for "Munich: The Price of Peace," a study of the appeasement of Hitler in the 1930s.Philip Levine received the poetry award for two collections, "Ashes" and "Seven Years from Somewhere." Elaine Pagels' "The Gnostic Gospels," an interpretation of some 2,000-year-old religious texts comparable to the Dead Sea Scrolls, received the criticism award.

A special award was designated for Flannery O'Connor's collected letters, "The Habit of Being," edited by Sally Fitzgerald.

Meeting at the Algonquin Hotel in New York, the 21 members of the NBCC board of directors announced their decisions at 7:30 p.m. after a three-hour voting session. The board, which represents nearly 200 book critics from around the country, chose from previously announced nominations, six in fiction and five each in the other categories. The criterion for selection was in chairman Eliot Fremont-Smith's words, "excellence, nothing else."

"The Year of the French," called by one reviewer "the finest historical novel by an American to appear in more than a decade," tells the story of a combined Irish-French military campaign against the British in 1798. The novel, written from several characters' viewpoints, strives for a balanced, compassionate depiction of all three nations involved in this doomed attempt at revolution. Flanagan, a professor of Irish history and literature, narrowly edged out Leslie Epstein and his controversial novel about the Holocaust, "King of the Jews." The other fiction nominations were Norman Mailer's "The Executioner's Song," William Styron's "Sophie's Choice," Philip Roth's "The Ghost Writer" and critic Elizabeth Hardwick's "Sleepless Nights."

The nonfiction award proved a close battle among three books: Tom Wolfe's "The Right Stuff," an account of the early days of the space program; Douglas Hofstadter's "Goedel, Escher, Bach," a freewheeling meditation on three mathematically minded genuises, and the winner, Telford Taylor's "Munich: The Price of Peace." Taylor, distinguished as both a historian and a lawyer (he was chief U.S. counsel at the Nuremberg trials), examines in detail the diplomatic maneuverings of Europe in response to the rise of Hitler. The book culminates in the ignominy of Neville Chamberlain's policy of appeasement.

Normally, the general membership of the NBCC nominates three books in each category and the board chooses two others. But this year no book of poetry received enough votes to be nominated by the general membership.

Consequently, the board itself chose all five nominations, deciding to allow Philip Levine's two books, "Ashes" and "Seven Years from Somewhere," to represent a single nomination. Not too surprisingly, Levine received the award, although Anthony Hecht's witty and elegant "The Venetian Vespers" was a close runner-up. Levine, who once studied with John Berryman, writes in a colloquial diction and has been much influenced by Spanish surrealists and by Kenneth Rexroth and the San Francisco poets. He has written that his "hope is to write poetry for people for whom there are no poems."

Few topics seem more esoteric than Gnosticism, a religion viewed as heresy by the early Christians. In "The Gnostic Gospels" Elaine Pagels elucidates the main tenets of the Gnostics, for whom the world was created by an evil Demiurge and who strive to return their "pneuma," or inmost breath, to the true, alien God. Among the matters Pagels discusses are the possibilities that the resurrection of Jesus was a figurative, mythological event and that there might be a female deity in contrast to the paternalistic God of the Old Testament.

Doubtless important, though unlikely to be popular, "The Gnostic Gospels" won out over Robert Alter and Carol Cosgrove's biography of the French novelist Stendhal, "A Lion for Love," and Frances FitzGerald's "America Revised," a study of history textbooks as a barometer of the way Americans think of themselves and their past.

The special award designated for Flannery O'Connor's "The Habit of Being" is the first given by the NBCC. O'Connor, a distinguished novelist and short-story writer who died in 1964, was also profoundly Christian and her letters reveal that her artistic and religious devotion were ultimately the same.

The NBCC awards have been watched with particular attention this year because of the controversy surrounding The American Book Awards, the successor to the recently abolished National Book Awards. Many writers and critics (and even certain publishers) have felt that TABA have demeaned the significance of literary awards by choosing to grant prizes in more than 30 categories. Many of the nation's best known writers have refused to allow their books to be considered for this prize. Consequently, the National Book Critics Circle selections have assumed an increasing importance.

The offical awards ceremonies honoring this year's winners will take place on Jan. 17 at the Time-Life Building in New York.