"Victory Over Japan" by Ellen Gilchrist has won the 1984 American Book Award for Fiction. Robert V. Remini's "Andrew Jackson and the Course of American Democracy 1833-1845, Vol. III," took the Non-Fiction prize, and "Stones for Ibarra" by Harriet Doerr was named the best First Work of Fiction.

The awards, sponsored by the Association of American Publishers, were announced last night at ceremonies in Astor Hall of the New York Public Library. Each of the three winning authors received $10,000; in addition, each of the 11 nominees for all three categories received a $1,000 award contributed by Simon & Schuster.

"Victory Over Japan," Gilchrist's third book, is a collection of 14 short stories catching ordinary people in surprise and crisis; "Stones for Ibarra," published when its author was 73, is a novel about a middle-class California couple who gamble everything to reopen an old mine in Mexico and find a violent clash of cultures.

This year's competition was the first under the new three-prize format. Last January, after a year of "debate and discussion," prize committee officials announced their decision to discard the previous complex system involving 27 separate awards, half to paperback reprints.

The American Book Awards (TABA) were created in 1980 to succeed the defunct National Book Awards. The NBA, administered by writers and critics, had earned literary esteem but little funding, and its detractors in the book industry complained that it ignored commercial concerns. TABA organizers enlarged the judging panels with representatives of publishing houses and libraries, and expanded the categories to include children's literature, cover design, illustration and paperback reprints and originals. Many former NBA winners protested, but publishers were very enthusiastic.

In 1983, more than 1,000 volumes were submitted for consideration, imposing "an unmanageably heavy administrative burden" and producing "an excessive level of cost," TABA officials said earlier this year. Moreover, the sheer number of prizes "tended seriously to diminish the value and prestige of any single award." So the number of categories was reduced to three. Only books written by American authors and published in the United States between Nov. 1, 1983 and Oct. 31, 1984 were eligible.

Jonathan Yardley of The Washington Post chaired this year's panel of fiction judges, which included authors Laurie Colwin and Leonard Michaels. In addition to "Victory Over Japan," they also nominated "Foreign Affairs" by Alison Lurie and "The Anatomy Lesson" by Philip Roth.

The nonfiction panel, headed by historian Arthur Schlesinger Jr., was made up of Harvard professor Daniel Aaron, biographers F. Scott Berg and Doris Kearns, and Charles Champlin, arts editor of The Los Angeles Times. Their nominations also included Eudora Welty's memoir, "One Writer's Beginnings," Ernst Pawel's biography of Franz Kafka, "The Nightmare of Reason," Howard M. Feinstein's "Becoming William James" and Richard Marius' "Thomas More, A Biography."

The judges for First Work of Fiction were critic and novelist Doris Grumbach (chairman), Digby Diehl of the Los Angeles Herald Examiner and author Alan Cheuse. Their runner-up nominees were "Tapping the Source" by Kem Nunn and "Edisto" by Padgett Powell.