A group of more than 40 former National Book Award winners and judges, including a Nobel laureate, this week called for a boycott of the NBA's successor, the 1980 American Book Awards. The new award, they say, will transform the prestigious, 30-year-old literary honors "into a rubber-stamp prize for best sellers."

In a joint letter to the literary press, the group protested the June decision by the Association of American Publishers to stop sponsoring the NBA, whose winners were selected by writers and critics. Under the new system, winners will be chosen by an "academy" composed of four groups: publishers; booksellers and distributors; librarians; and authors, critics and scholars.

"In other words," the letter said, "when it comes to picking the winners, writers and critics will have fewer than one vote in four."

As a result, the group argued, the new rules would give the awards an unwanted commercial aspect. "Evidently, writers and critics are undependable types, who cannot be trusted to know that the best book is that which sells the most copies," the letter said.

Ronald Bush, president of Pocketbooks and co-chairman of the board of directors of the American Book Awards, yesterday defended the new system: "We plan to have 15 judges in each category who will read a lot more than the three NBA judges in each category could. It's sheer mathematics." Bush said that the ABA was not intended to replace the NBA. "It just turns out that the NBA will cease and the ABA will continue," he said.

Yesterday several NBA winners explained why they had joined in the protest. Tim O'Brien, whose Vietnam novel "Going After Cacciato" won this year's fiction prize, said the publisher's association is "trying to Hollywoodize the book award." The decision to expand the number of awards from seven to 25 - including winners for cookbooks, science fiction, Westerns and hobbies - is "the equivalent of giving an Academy Award for the best Kung Fu-Bruce Lee movie," O'Brien said.

Bernard Malamud, 1959 NBA winner for "The Magic Barrel," said yesterday that he would not submit any future book of his to the ABA process. All the other cosigners of the letter - including Donald Barthelme, Saul Bellow, Malcolm Cowley, Elizabeth Hardwick, Irving Howe, Alfred Kazin, Howard Nemerov, Joyce Carol Oates, Walker Percy, Philip Roth and Susan Sontag - said they would refuse to let their works be submitted to the ABA process.

"You can't choose the best," Malamud said, "when you don't have the best to choose from."

William Styron, whose novel, "Sophie's Choice," is now number one on the bestseller list, has also decided to ask his publisher not to submit his books for the ABA.

"I don't think it's a very complicated issue. Everybody felt the new award was heavily weighted for media hype and unfair to books without publicity," he said.

The author of the letter, former NBA judge Alison Lurie, expressed the hope that the AAP "would see the errors of their ways and go back to the old system. If all these people boycott them - and almost everyone is supporting us - either the awards won't continue as planned or they won't have any literary standing."

Bush, however, maintained that the award would continue to carry prestige. "The assumption that professionals who work with the very same authors and critics are incompetent to judge quality is an assumption I can't quite understand. Paperback publishers read five times as many books as hardcover editors," he said, and "it would be far better for the industry if we all joined together to make a better program."

The ABA board of directors, in a statement responding to the protest letter, said, in part: "In no event will publishers or nominating committee members or voters be instructed to select books which have sold well. Rather they will be advised to choose the best books in each category."

The protesting group also criticized the new award organization for requiring payment of a fee for each book submitted "rather than asking their judges to make selections from all books in their category published in a given year, as before." They see this requirement as an attempt "to discourage many publishers from submitting their less commercial titles, and perhaps to discourage some small presses from submitting any titles at all."

The ABA board replied that such charges are "speculative" and that "to insure that no worthy book will be overlooked, the board of directors many months ago decided that titles could also be submitted by the board to the nominating committees."