Alex Haley, whose family history, "Roots," is one of the best-selling hardcover books of all time, is suing his publisher, Doubleday, for $5 million punitive damages.

The 35-page complaint, filed late Wednesday in Los Angeles Superior Court, charges, in effect, that Doubleday undercut the sales of the hardcover book in order to profit from increased sales on the paperback versions to be published by its subsidiaries.

He complains that the book was not advertised enough, that he himself had to arrange his own speaking engagements, that copies often were not available in bookstores he and his friends checked at random.

But the complaint is about a book which now had 1.6 million copies in print, has been the subject of an eight-day, prime-time television series, and is one of the most talked-about and written-about books of the decade.

Haley's position is that the sales could have more than doubled if promotion, distribution and the timing of paperback editions had been handled differently.

Haley hopes to get compensatory damages by establishing in court what the sales of the book might have been if Doubleday had promoted sales of the book as he thought proper.

The unusual publishing situation of "Roots" arises from Doubleday's having sold the paperback rights to the Dell Publishing Company in 1967, and the Doubleday's having bought the Dell company in 1976.

Under Doubleday's original contract with Dell, Doubleday would receive 10 per cent of the paperback receipts, which it would split with Haley. There were also provisions, according to Haley's lawyers, preventing an appearance of a paperback edition until 18 months of after the publication date, which was Oct. 1, 1976.

Haley now charges that Doubleday instead renegotiated to advance the combined companies' interests - the Doubleday-Dell share of receipts now being 95 per cent.

Haley charges that Doubleday should have renegotiated the Dell contract in accordance with provisions applying to a best-seller. In contracts between Haley and Doubleday, negotiated in 1974, mention was made of "an adjustment of the terms of the paperback, licensed in accordance with custom and usage in the publishing industry."

Doubleday's editor-in-chief, Stewart Richardson, said yesterday that no new contract between Doubleday and Dell had been made.

According to Haley's complaint, a new agreement was made which provides for a "class" paperback, to be sold for $6.95, to be issued this September by Dell's imprint, Delta, and for a "mass" paperback, at $2.95, to be issued by Dell in January, 1978.

The very announcement of such intentions has already had an effect on hardcover sales, Haley charges, and the appearance of the books in paper will drastically cut back hardcover sales.

Haley gets 15 per cent or $1.87, for each $12.50 hardcover; and would get 5 per cent on each paperback, which would mean 35 cents a copy for the Delta book and 15 cents for the Dell version. Doubleday would not only get a larger per cent of the soft-cover profits, he charges, but would save money on the publishing costs.

He alleges that Doubleday failed to promote the book in a way that was commensurate with it sales, and instead of distributing sufficient copies to meet the demand at its height, spread the word about forthcoming paperback editions.

Richardson's reply, which he characterized as "more in sorrow than anger." was that "we spent a ton of money on it, advertising and promoting it."

But, he added, "for the first time that I can recall in the history of the company, we were unable to supply the demand from our own printing facilities and we went to an outside plant to print more."

Haley's complaint alleges that Doubleday had been "forewarned" by him of the huge demand that would occur when the television series appeared.

He also charges that "at his own expense," he had "spent over 11 years promoting and exploiting said work by delivering hundreds of lectures on the subject matter of 'Roots'" as well as television shows and interviews.

It says that Doubleday "is well aware that public exposure of a literary work through advertisements, announcements and similar promotional activities on radio, television and in the published media is necessary to achieve the highest possible volume of sales" but that the company "failed to adequately exmploit 'Roots'" in such as manner.

Haley's original was signed with Doubleday in 1964. Deadlines for submission of his manuscript were extended several times before the final version of the book was completed early in 1976. Meanwhile, Haley had supported himself and his research with various advances and the lecture fees from his prewriting promotion work.