The increasingly controversial American Book Awards will take a different shape next year, the Association of American Publishers board of directors agreed Thursday. Although some members were reportedly in favor of abolishing the costly and complicated prize competition, "we've decided to go ahead with a scaled-down version," said AAP chairman Brooks Thomas, president and chief executive officer of Harper & Row. "The idea of modeling it on the Academy Awards was a mistake."

At a July 12 meeting of the board's executive committee, members reportedly had split evenly over the question of whether to continue the TABA awards, which permit unlimited submissions from publishers. The 1983 event cost about $250,000 for 112 judges to select 27 categories of winners from about 1,500 submitted titles. Each winner receives $1,000 and a small sculpture.

At the Thursday meeting, Thomas said, the board reached a "consensus" that the format had become unwieldy. But it left open the questions of "the number of categories and what they will be, as well as how the bills will be paid and by which constituencies" within the AAP's 300 members: "whether the cost should be borne across the board, or focused" on individual divisions, such as trade publishing, which benefit most.

"We'll re-address that question," AAP president Townsend Hoopes said, "in the context of our ongoing priorities review"--a member survey which should be completed by the next board meeting Nov. 1.

TABA was created four years ago to replace the artistically esteemed but commercially ineffectual National Book Awards. Many booksellers and publishers had become reluctant to sponsor the NBA, complaining that its seven categories of winners were too few and too "literary" and that its judging panels of writers and critics did not reflect the tastes of the ordinary reader.

So in 1979 TABA expanded the format to 13 categories--including paperbacks, reprints, children's books and science fiction--and gave majority control of the judging to booksellers, librarians and publishers. Commercial aspects of the change angered many former NBA winners, more than 40 of whom agreed to boycott the new award by forbidding their publishers to nominate their books. And that was well before the 1982 "total marketing program," in which bookstores gave customers TABA ballots to send in with their choices marked. The ballots were not seen by the judges, but those which predicted the winners correctly were entered in a prize drawing.

As TABA grew more accommodating, its burgeoning complexity and disappointing promotional value grew more irksome to many book industry officials. Last June, Hoopes prepared a report citing the principal problems. Among them:

* Incompatiblility between the two AAP divisions which control the awards--general publishing's interest in literary quality, and the paperback division's emphasis on boosting sales.

Veto power by either side over the board's decisions.

* Concern that the April awards date did not promote sales, since many of the previous year's spring and summer titles are out of stores by then.

* The high price and annoyance of processing so many nominated titles (which must be shipped twice--once to a central point and then to the judges in order to keep their identities secret).

* The difficulty of recruiting the necessary one-fourth of TABA's budget from private, corporate and other sponsors because TABA contributions--unlike those to the NBA--do not qualify legally as charitable gifts.

Hoopes' report recommended reconstituting the awards under AAP direction and funding; concentrating on rewarding quality; reducing the number of categories to 10 or fewer with publishers restricted to no more than three submissions for each, paperback reprints disqualified and a single award for paperback originals; eliminating anonymity for judges; moving the date to November, and holding the cost to about $100,000.

"It shouldn't cost more than that," said TABA executive director Barbara Prete. "The reason it has been so high is that you've had these two divisions with opposing goals both calling the shots--so everybody's wishes were accommodated." That created a climate, she said, in which "everybody's got a suggestion for a category and nobody should be offended." Removing the funding and control from general publishing and paperbacks and distributing it among AAP's 300 members would not cost each house much, she says, "but that's the rub. Publishers don't want to spend that kind of money. They'd rather take out an ad and sell a book."