The National Book Critics Circle yesterday named John Updike's "Rabbit Is Rich" the outstanding novel of 1981 and awarded the general nonfiction prize to Harvard biologist Stephen Jay Gould for "The Mismeasure of Man."

"The Virgil Thomson Reader," a selection of writings by the 85-year-old composer and music critic, won the award for criticism; and A.R. Ammons' collection, "A Coast of Trees," took the poetry prize.

The winners of the seventh annual awards were chosen from a group of 20 nominees -- five each in the four categories, and this year all men -- selected by the NBCC's 300 members. The final selections were made yesterday by 21 members of the NBCC board of directors meeting at New York's Algonquin Hotel. The voting -- which in some previous years has been marked by protracted acrimony -- took 3 1/2 hours and was generally harmonious.

The debate over the fiction prize immediately devolved into a contest between two Knopf books: Updike's third and most exuberant novel about the domestic aggravations and spiritual yearnings of Harry "Rabbit" Angstrom and "A Flag for Sunrise," Robert Stone's grim and ambitious fable of guilt and redemption set against the background of a South American revolution. Several of the judges were adamantly opposed to the nihilistic tone of Stone's work, and were prepared to argue indefinitely against it. But gradually the Stone supporters gave way, leaving Updike a clear winner. Also nominated were "Riddley Walker" by Russell Hoban (Summit); "Sixty Stories" by Donald Barthelme (Putnam); and "The Men's Club" by Leonard Michaels (Farrar, Straus and Giroux).

After the early balloting, there were two principal contenders in general nonfiction. One was "The Mismeasure of Man" (Norton), Gould's study of how theories of genetic determinism have been influenced by social bias over the past 200 years, in which he concludes that "Science must be understood as a social phenomenon." The other was "National Defense" (Random House) by James Fallows, Washington editor of The Atlantic Monthly and former aide to President Jimmy Carter. Fallows criticizes "gold plating" in weapons contracts and is highly skeptical of the notion that more spending will actually produce a stronger defense. The other nominees were "The Sage of Monticello" by Dumas Malone (Little, Brown); "Forms of Talk" by Erving Goffman (University of Pennsylvania); and "No Place of Grace: Anti-Modernism and the Transformation of American Culture" by Jackson Lears (Pantheon).

In the criticism category, Houghton Mifflin's 582-page collection of Thomson's work dominated the voting from the outset, with many judges praising the former critic for the New York Herald Tribune. Thomson's work, which spans several decades, is simple, clear and often wryly humorous: "At the back of every conductor's mind is a desire to make his orchestra produce a louder noise than anyone else's . . . he goes into a brief convulsion at that point. The convulsion is useful to the conductor, because it prevents his hearing what the orchestra really sounds like while his fit is on." The nearest runner-up in the voting was science-writer Martin Gardiner's collected articles, "Science: Good, Bad and Bogus" (Prometheus), which debunks the miraculous claims and popular reputations of extrasensory perception, clairvoyance and telekinesis, among others. Other nominees were "The Geography of the Imagination" by Guy Davenport (North Point Press), "Early Auden" by Edward Mendelson (Viking) and "Boccaccio" by Thomas G. Bergin (Viking).

The poetry debate proved the most heated, and eventually centered around three titles: "The Revisionist," a collection by Douglas Crase (Little, Brown); "Brotherly Love," an epic poem about William Penn by Daniel Hoffman (Random/Vintage) and Ammons' "A Coast of Trees" (Norton). The judges split into factions, with generalists favoring Crase or Hoffman and poetry aficionados supporting Ammons. After repeated balloting, Ammons' discursive and rural poems, in which natural images become metaphors for forms of thought ("The very longest swell in the ocean,/I suspect,/carries the deepest memory"), emerged as a consensus choice. Also nominated were "What Manner of Beast" by Donald Finkel (Atheneum) and "For the Sleepwalkers" by Edward Hirsch (Knopf).

The awards will be presented to both winners and nominees on Jan. 28 at a ceremony and reception in the New York Public Library.