On Sunday, Oct. 3, when the New York Times Book Review premiered its redesigned bestseller pages, Patricia Cornwell's Trace topped the fiction list, and Kitty Kelley's The Family was #1 in nonfiction. The fine print below the table explained that the rankings reflected sales for the week ending Sept. 18 -- a period about two weeks earlier. So by the time readers saw the paper, the list no longer reflected reality: Trace and The Family had both actually fallen to #2, replaced at the top by Stephen King's newest installment of The Dark Tower and, in nonfiction, by Jon Stewart's America (The Book). But a N.Y. Times reader wouldn't have known it. Lower-ranking slots revealed the lag's effect even more strikingly: Some titles on that Oct. 3 list (such as David Sedaris's latest essay collection) had fallen off as early as Sept. 25.
This lag between compilation and publication is nothing new to book publishers, many of whom pay the N.Y. Times a $1,140 annual subscription fee to see the list 10 days before publication date. And in a recent study, Stanford business professor Alan Sorensen exploited such time lags to test the effect of appearing on the N.Y. Times list upon subsequent sales of hardcover fiction. (The list does affect sales, though much less than expected.) But looking at N.Y. Times bestsellers alone wouldn't prove that any week-to-week sales changes were caused by the list itself. To answer the causal question, Sorensen needed a comparison group: books that sold well but weren't ranked on the Times list. So he looked at data from Nielsen BookScan, the point-of-sale monitoring service from Nielsen Entertainment. Sorensen's study, which is the first to publicly compare a well-known bestseller list with Nielsen's figures, raises another question: If Nielsen BookScan data are more accurate and more timely, why do newspapers and magazines continue to go through the effort of compiling their own lists?
This article is based on numerous interviews with industry experts and representatives from Publishers Weekly, the N.Y. Times, USA Today and the Los Angeles Times. In the course of these interviews, other questions emerged, suggesting that much more than timeliness is at issue.
A Matter of National Security?
Jim King, vice president and general manager of Nielsen BookScan, believes that BookScan's methodology is sounder because it compiles its charts from point-of-sale data and because it can account for nearly 70 percent of all book sales in the United States. "Point-of-sale" means that the information is taken directly from cash registers. Each week, participating retailers -- which include online booksellers such as Amazon.com, chains such as Borders, Waldenbooks and Barnes & Noble, independents such as Politics and Prose, and some discounters such as Costco and Target -- send Nielsen a sales file of data scanned during purchases. This information, which includes each sale's location, flows into Nielsen's database. (To track Internet sales, Nielsen stores the zip code of each sale's destination.)
Periodicals that create their own national lists, on the other hand, typically collect data from only a sample of stores that they believe represent all domestic booksellers, and each tries to infer total sales rankings from its own sample. (The Los Angeles Times publishes a regional list from a sample of about 30 Southern California booksellers.) King sees that as a very different process from what BookScan does: "We don't just call up a random group of bookstores and ask how many copies of books you sold." In sampling, true randomness can be a strength, but each periodical's specific mix of reporting booksellers is decidedly non-random, determined by a combination of editors' and booksellers' choices. At Publishers Weekly, executive editor Daisy Maryles -- who has been working on the trade weekly's bestseller lists for almost 30 years -- says she is still trying to crack Wal-Mart as a source, as is, for that matter, Jim King at Nielsen. Steve Wasserman, the Book Review editor at the Los Angeles Times, doesn't even pretend that his staff's process yields reliable results. "It's a deeply unscientific -- one is almost tempted to call it whimsical -- compilation, which has a veneer of a certain kind of science," he says.
Some of what Wasserman says of the L.A. Times list could be true of other lists as well. "I don't think there's 100 percent reliability from any source, including BookScan," says Irwyn Applebaum, president and publisher of the Bantam Dell Publishing Group, a division of Random House. BookScan comes the closest to that goal, Applebaum says, by gathering over-the-counter sales from the largest number of retailers, but it is "by no means complete." Missing are sales not only from Wal-Mart and Sam's Club, but also from drugstores, supermarkets, the smaller independents, and retailers -- like Williams-Sonoma -- that sell a few cookbooks alongside their pots and pans.
If incomplete information is a fact of life, what's more troubling is that some of the listmakers could be working with a biased sample, one that systematically omits certain types of sources and gives inordinate weight to others. "It's a matter of who is feeding each list," says Carolyn Reidy, president of Simon & Schuster's Adult Publishing Group. From years of comparing the lists and hearing from booksellers, she has come to believe that the N.Y. Times list is weighted more to the "old-fashioned bookstore kind of market," whereas USA Today seems skewed toward mass merchants and wholesalers who sell to non-bookstore outlets.
But because the listmakers don't release their statistical methodology, such bias is as unverifiable as listmakers' denials that bias exists. Fordham University business professor Albert Greco, who consults for the Book Industry Study Group, says, "If they were in math class, the instructor would say, 'What are they doing?' "
To varying degrees, all the publications refuse to "show their work," especially specifics about weighting schemes and data sources. Publishers Weekly editors spoke candidly for over an hour about their process, but ultimately Maryles said there is no set formula for yielding the rankings. Furthermore, she says, "We don't tell people whom we call; we don't even tell people how many calls we make." In its aim at transparency, USA Today, on the other hand, publishes the names of some of its 4,700 sources -- the only periodical surveyed to do so -- but the paper's database editor, Anthony DeBarros, says that revealing the exact weighting scheme could somehow help readers deduce the number of titles sold by each contributing bookseller. Richard Meislin, editor of News Surveys at the N.Y. Times, cites similar reasons for staying closed-mouthed, not wishing to jeopardize the confidentiality of what he calls the "intelligence network" of booksellers on which the well-regarded list depends. The Times list draws on sales "at almost 4,000 bookstores plus wholesalers serving 50,000 other retailers," but Meislin wouldn't reveal what percentage of total sales that sample represents. Even the titles of books that showed aberrant spikes in some stores are confidential information.
Such secrecy seems absurd to the L.A. Times's Wasserman, who says of his paper that "we're not mysterious about how we arrive at this alchemy -- nor do we regard it as a matter of national security." But he, too, chose not to reveal names of contributing stores.
Some secrecy makes sense, of course. If readers knew which independent bookstores were sampled -- particularly if these smaller stores receive extra weight in a calculation of rankings -- authors or publishers might be tempted to buy up books from those sources in an attempt to boost their rankings on an influential list. It's been tried before and, says Greco, may be why the N.Y. Times is particularly wary of bulk orders, marking such bestselling titles with a dagger symbol. (Nielsen Book-Scan marks its lists for the same information, too.) "The ability to put 'New York Times Bestseller' [on a book cover] is a valuable thing, and we try to keep the list as honest and unmanipulated as we can," Meislin explains.
To Each His Own
To the N.Y. Times's Meislin, the answer to why newspapers create their own lists rather than buying them from BookScan seems self-evident. "The question is sort of like, 'Given the existence of the Associated Press, why does the N.Y. Times or The Washington Post continue to field its own reporters?' " Indeed, many regional newspapers and Entertainment Weekly subscribe to lists compiled by the N.Y. Times, USA Today or Publishers Weekly. But outsourcing a large weekly survey isn't quite the same as taking a story off the wire. In fact, The Washington Post decided last year to start buying the data for a specially tailored regional bestseller list from Nielsen BookScan. Marie Arana, editor of Book World, says, "We feel we are providing readers with superior reporting: In the Washington area alone, BookScan pulls our lists from hundreds more sources than we were able to access previously."
But other publications prefer to do it their way. "To Nielsen BookScan, as far as I understand, a book is a book is a book," Meislin says, whereas the N.Y. Times makes some editorial judgments -- most noticeably in its decision to put certain nonfiction titles into the periodic "Advice, How-To and Miscellaneous" category, which cuts off at five titles each in hardcover and paperback. This decision gives a title like Maureen Dowd's Bushworld at least as much prominence as The South Beach Diet, although the former may sell far fewer copies. And it's possible for a how-to title that sells phenomenally well not to make the list at all.
USA Today's list, which ranks all bestselling titles in one take, reflects a more populist vision of showing "America's book-buying habits, what is truly selling each week," whether the author is Philip Roth or Dr. Phil McGraw, Nora Roberts or Lemony Snicket. But nothing says editors can't reshape the Nielsen data as they wish. The Post, for example, receives BookScan's top 50 list for the Washington area and divides it into a general list and a monthly advice/self-help list.